How to read this page

Scope: how to model speech-to-speech (S2S) systems, treating the reasoning LLM as a downstream component. It covers newer-generation models and strategies in ASR and speech generation, plus a system design for Synthesia.

Each section is one topic, at most two printed pages, and ends with a "Say this in the interview" line. The cheat sheet in the last chapter is the one-page recall aid.

Grounding legend

Every model detail and number on this page is traced to a research paper (cited inline as arXiv IDs) or to the vendor's own page (linked inline). Numbers that the authors or vendor measured on their own systems are phrased as "the authors report".

• Reported A number from the authors' or vendor's own first-party source.
• Inference Design opinion, not fact.

Three traps

The S2S design space in one picture

Every speech-to-speech system answers the same five questions. The model zoo looks large only because the answers combine in many ways.

The core tension

Speech encoders emit frames at 50 Hz (Whisper, as used in Voxtral; arXiv 2507.13264), while text is far sparser. Every decision above is a way to give the LLM a sequence it can model at text-like cost. The goal is to do that without losing prosody, speaker identity or timing.

Why the LLM can be "downstream"

If the backbone sees speech as embeddings and emits tokens a separate renderer turns into audio, the reasoning model can be swapped, frozen or fine-tuned lightly. Freeze-Omni (arXiv 2411.00774) takes this to the limit by freezing the LLM.

The open problem

Keeping text-level intelligence once the model must speak. URO-Bench reports that open-source spoken dialogue models lag behind their backbone LLMs on instruction following and suffer from catastrophic forgetting (arXiv 2502.17810).

Speech input for LLMs

Three ways to feed speech to an LLM: continuous encoder embeddings, discrete semantic tokens, or codec tokens. Most models use the first.

Per-model paths

Why 12.5 Hz converges

Whisper emits 50 Hz frames. Voxtral notes that 30 minutes of audio at 50 Hz is a sequence of 90k tokens, which is why it adds a 4x downsampling adapter (arXiv 2507.13264). Moshi's Mimi codec also runs at 12.5 Hz (arXiv 2410.00037).

Trade-offs

• Continuous: richest acoustic and paralinguistic signal, easy to train as ASR, but the output side needs a separate path.
• Discrete: one vocabulary for input and output, enables self-supervised next-token training, loses detail.
• Codec: the model hears and speaks the same tokens, but multi-codebook streams cost more per frame. Inference This suits duplex.
• Inference Downsampling to 10-12.5 Hz is the main sequence-length lever. It costs fine acoustic detail, which motivates hybrids like Kimi-Audio.

ASR with speech LLMs: offline and streaming

ASR is one more speech-to-text task on the encoder, projector and LLM stack. The design questions are how much to train, and how to stream.

Training recipes

• Freezing the LLM is the direct guard against forgetting. Granite-speech states that in text mode it calls the underlying LLM without LoRA, which preserves its text capabilities.
• Canary-Qwen's LLM mode takes a text transcript only, not raw audio (NVIDIA model card).

Streaming approaches

Inference Chunk size and lookahead trade latency against WER.

Multilingual scale

Omnilingual ASR (Meta) covers over 1,600 languages, in models from 300M to 7B parameters (arXiv 2511.09690). Qwen3-ASR covers 52 languages and dialects (arXiv 2601.21337).

Leaderboard state

The Open ASR Leaderboard compares 86 systems on 12 datasets, with English short- and long-form and multilingual tracks (arXiv 2510.06961). Its authors observe that Conformer encoders paired with Transformer decoders reach the best average WER, while CTC and TDT decoders give the best RTFx.

Open question and scope

• Not covered here: transducer plus LLM hybrids and CTC-prefix joint decoding.

Whisper and the Whisper encoder

Whisper is an encoder-decoder Transformer trained on 680,000 hours of weakly supervised audio. Its encoder became the default front end for speech LLMs. Source: arXiv 2212.04356 unless stated.

Architecture

• Confirmed Input is 16 kHz audio turned into an 80-channel log-magnitude mel spectrogram (25 ms window, 10 ms stride), in 30 s segments.
• Confirmed The stem is two conv layers (width 3, GELU); the second has stride 2, so one encoder frame covers 20 ms.
• Derived: the 10 ms hop gives 100 mel frames per second (100 Hz). The stride-2 conv halves that to one frame per 20 ms (50 Hz). A 30 s window is therefore 30 s / 10 ms = 3,000 mel frames and 3,000 / 2 = 1,500 encoder frames. Short audio is padded to 30 s, so compute is paid for the full window.

Rule (derived arithmetic):

• large-v3: same architecture, 128 mel bins instead of 80, and a Cantonese token. Reported OpenAI reports a 10% to 20% error reduction over large-v2 (OpenAI model card).
• large-v3-turbo: a fine-tuned, pruned large-v3 with the decoder cut from 32 to 4 layers (OpenAI model card). The card lists it at 809M parameters.

Multitask token format

The decoder prompt carries the task as tokens: start-of-transcript, language, task (transcribe or translate), then timestamps or no-timestamps. One model therefore does ASR, X-to-English translation and language ID. Timestamps are quantised to 20 ms.

Training data and recipe

• Filtering: drop machine-generated transcripts (all caps or all lower, no punctuation). Compare audio language ID (VoxLingua107-trained) with CLD2 on the transcript and drop mismatches; English transcripts of other-language audio are kept as translation data.
• Optimiser: AdamW, gradient-norm clipping, linear decay after 2,048 warmup steps, batch 256, FP16 with activation checkpointing, 2 to 3 passes over the data.
• Augmentation: none for the first models. Large-v2 trained 2.5x more epochs with SpecAugment, stochastic depth and BPE dropout.

Long-form decoding

Whisper slides a 30 s window over long audio and decodes with heuristics (paper section 4.5).

• Beam search with beam 5.
• Temperature fallback from 0, in steps of 0.2 up to 1.0, when average log-probability is below -1 or the gzip compression ratio exceeds 2.4.
• Condition on the previous window's text only when temperature is below 0.5.
• No-speech threshold 0.6 to skip silent windows.

Limitations

• The model card warns that predictions may include text that was not spoken (hallucination) and that the sequence-to-sequence architecture is prone to repetitive text. It also states that the models cannot be used for real-time transcription out of the box.
• The fixed 30 s window, non-causal encoder and autoregressive beam decoder make streaming hard.

The encoder as a speech-LLM front end

Inference Three issues recur. Padding to 30 s wastes compute on short turns. 50 Hz is too long a sequence for an LLM, so every system adds an adapter (50 Hz / 4 = 12.5 Hz). Bidirectional attention blocks streaming, so teams either make the encoder causal (GLM-4-Voice) or replace it (AuT).

Making the Whisper encoder stream

Whisper does not stream for three reasons.

• The encoder takes a fixed 30 s window (arXiv 2212.04356), and Voxtral describes it as a fixed receptive field of 30 s (arXiv 2507.13264).
• Streaming wrappers pad every buffer to the full context length, so each partial result pays for 30 s (arXiv 2506.12154).
• Encoder attention is bidirectional, so every frame sees later audio (GLM-4-Voice replaces it with block-causal attention, arXiv 2412.02612). The model card states the models cannot do real-time transcription out of the box.

Four families fix this, with different cost.

• Causal fine-tuning. CarelessWhisper fine-tunes Whisper with LoRA so each encoder frame attends only to its own chunk and earlier chunks, and it caches encoder keys and values instead of re-encoding padded buffers (arXiv 2508.12301). It trains on 960 h of LibriSpeech with chunk sizes of 40 to 300 ms. The U2-style variant puts a CTC head on the Whisper encoder, trains with dynamic chunk masks (0.1 to 1.0 s) on inputs shorter than 30 s, and rescores with the original decoder (arXiv 2506.12154). GLM-4-Voice makes the encoder causal inside its tokenizer: causal convolution before the Transformer and block-causal attention, then pooling and vector quantisation give 12.5 Hz tokens (arXiv 2412.02612).
• Replace the encoder. Qwen3-Omni trains AuT (Audio Transformer) from scratch on 20 million hours of supervised audio, with 8x Conv2D downsampling to 12.5 Hz and block-wise window attention for real-time prefill caching (arXiv 2509.17765). Qwen3-ASR reuses AuT with a dynamic flash-attention window of 1 s to 8 s, so one model serves streaming (short chunks) and offline (long queries) (arXiv 2601.21337).
• Wrap the offline model. whisper_streaming re-runs Whisper on a growing buffer and commits words once consecutive runs agree (LocalAgreement), at 3.3 s latency (arXiv 2307.14743). Simul-Whisper uses the cross-attention alignment to guide decoding on chunks, with a truncation detector for cut words (arXiv 2406.10052). The U2 authors note that wrappers reprocess the same audio and leave a training-inference mismatch (arXiv 2506.12154).

Streaming audio encoders beyond Whisper

Most streaming encoders do one of two things: restrict attention to a causal or windowed context, or cache past activations.

• Qwen3-ASR numbers: the paper reports 2.69 average WER offline vs 3.33 streaming for the 1.7B model, and 3.48 vs 4.40 for 0.6B, with streaming run at a 2 s chunk size (arXiv 2601.21337). It also reports an average time to first token as low as 92 ms for the 0.6B model.
• DSM-ASR: feeds Mimi tokens (12.5 Hz) and delays the text stream by a default 2.5 s, with timestamp precision of 80 ms (arXiv 2509.08753).
• Cache-aware Conformer: with a regular look-ahead, latency compounds with depth (1,360 ms for 17 layers and one token of look-ahead); chunk-aware look-ahead does not. On LibriSpeech test-other with an RNNT decoder, the cache-aware model reaches 6.3% WER at 1,360 ms, against 11.3% for buffered inference at 2,000 ms (arXiv 2312.17279).
• The model card lists a 120M-parameter cache-aware FastConformer-RNNT with 17 layers and latency of 80 to 160 ms.

Ecosystem

Interview insights

Codecs and speech tokenizers

The tokenizer fixes the sequence length the LM sees and the ceiling on audio quality. Four families matter.

Qwen3-TTS ships two tokenizers: 25 Hz single-codebook and 12 Hz multi-codebook (arXiv 2601.15621).

Trade-offs

Evidence on low frame rates

A controlled frame-rate ablation reproduces a quality cliff at 6.25 Hz. Its authors trace the cliff to the training configuration (fixed clip duration gives too few tokens at low rates), not to a fundamental barrier. After the fix, WER degrades smoothly with phonemic load down to 3.1 Hz and 1.6 Hz (arXiv 2606.16969). Another paper pools X-Codec-2.0 from 50 Hz to a 25 Hz latent rate and reports a 0.29 UTMOSv2-based MOS gain (arXiv 2601.20185).

Generation paradigms

There are four ways to generate speech from text or from an LM state. The dominant high-quality design is the hybrid.

AR codec LMs

• Llasa: a single Llama-style Transformer over X-Codec2 tokens, 1B/3B/8B, with inference-time search using speech-understanding verifiers (arXiv 2502.04128).
• Spark-TTS: Qwen2.5 LLM plus BiCodec (arXiv 2503.01710).
• CosyVoice 2: an LLM predicts speech tokens, then a chunk-aware causal flow-matching model decodes them (arXiv 2412.10117).
• Newer entries: IndexTTS2 (arXiv 2506.21619), Step-Audio-EditX (LLM-based audio editing with reinforcement learning; arXiv 2511.03601), Higgs Audio v2 (pretrained on over 10 million hours of audio, per the Boson AI repo).

Multi-codebook handling

An RVQ frame has several codes, so the model needs a trick.

• Sesame CSM: a Llama backbone models the zeroth Mimi codebook from interleaved text and audio, and a smaller audio decoder with a linear head per codebook models the rest; Mimi runs at 12.5 Hz (Sesame blog).
• Moshi: $K = 2Q + 1 = 17$ streams per step with $Q = 8$ codebooks (8 user audio, 8 own audio, 1 text). Its RQ-Transformer splits temporal and per-codebook (depth) modelling (arXiv 2410.00037).
• Qwen3-Omni's Talker uses an MTP module to emit codec frames (arXiv 2509.17765).

NAR notes

SoundStorm uses bidirectional attention and confidence-based parallel decoding to generate codec tokens, and the authors report 30 s of audio in 0.5 s on a TPU-v4 (arXiv 2305.09636). MaskGCT is two-stage (text to semantic tokens, then semantic to acoustic tokens), mask-and-predict in both, with no explicit alignment or phone-level duration model, trained on 100K h (arXiv 2409.00750).

The "LM for content, flow for acoustics" split

• Inference Why the LM: text streaming, interleaving with text, reuse of LLM serving stacks, in-context learning.
• Inference Why flow: it can restore acoustic detail lost to quantization, and CosyVoice 2 shows it can decode chunk by chunk (arXiv 2412.10117).

Diffusion and flow matching for speech

Flow matching is the objective behind many recent TTS systems. It plays two roles: fully NAR TTS, and the acoustic decoder after an AR LM.

Diffusion objective (DDPM)

The forward process mixes clean data $x_0$ with Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$. The network $\epsilon_\theta$ learns to predict that noise from the noisy input and the timestep $t$.

Here $\bar\alpha_t$ is the cumulative noise schedule, so it fixes how much signal survives at step $t$. Sampling runs the learned denoiser backwards from pure noise.

This is the noise-prediction objective of the DDPM paper (arXiv 2006.11239).

Conditional flow matching objective

Flow matching regresses a network onto a velocity field. With a linear (optimal-transport) path, $x_0$ is noise and $x_1$ is data.

The target velocity $x_1 - x_0$ is constant along the path, so training is plain regression. There is no noise schedule and no SNR weighting.

Sampling integrates the learned ODE from noise to data:

The linear path and velocity regression follow Flow Matching with optimal-transport paths (arXiv 2210.02747).

Classifier-free guidance (CFG)

CFG pushes the output toward the condition $c$ by extrapolating away from an unconditional prediction $\varnothing$. The guidance scale $w$ sets the strength.

The cost is two forward passes per sampling step, so CFG doubles inference compute. The same form applies to a diffusion network by swapping $v_\theta$ for $\epsilon_\theta$.

Classifier-free guidance was introduced for diffusion models (arXiv 2207.12598); the form above applies it to a velocity network.

In speech, ZipVoice targets the CFG cost (arXiv 2506.13053). Lipman et al. show OT paths train and sample faster than diffusion paths (arXiv 2210.02747).

Where it sits

Counterexamples and traps

ASR has little flow work. Drax is the clearest, using discrete flow matching for NAR ASR (arXiv 2510.04162).

Continuous autoregression and next-token diffusion

A smaller, newer branch drops discrete tokens. An AR model predicts continuous latents, and a small diffusion or flow head samples each next frame or patch.

Inference The two lists below are design analysis, not results from the papers.

Streaming and low-latency generation

Streaming TTS needs a causal, low-frame-rate codec on the AR side, and a chunkable, few-step decoder on the acoustic side.

First-packet numbers

Latency levers

MeanFlow's headline FID 3.43 is an ImageNet one-step result (arXiv 2505.13447). IntMeanFlow reports 1-NFE token-to-spectrogram and 3-NFE text-to-spectrogram inference for speech (arXiv 2510.07979).

S2S architectures with the LLM as a downstream component

Six families, ordered by how tightly speech is fused into the LLM. The first three are the most conservative designs; the last three fuse speech into the LLM most tightly.

The six families

Key facts per model

• LLaMA-Omni 2: 0.5B to 14B on Qwen2.5, Whisper-large-v3 encoder, a streaming speech decoder inspired by CosyVoice 2, trained on 200K multi-turn dialogues; the authors report beating GLM-4-Voice (arXiv 2505.02625).
• Freeze-Omni: frozen LLM, so the authors report speech-mode intelligence at the same level as the text mode of the backbone, plus chunk-wise state prediction for duplex dialogue (arXiv 2411.00774).
• Qwen3-Omni: text in 119 languages, speech understanding in 19, speech generation in 10 (arXiv 2509.17765).
• Qwen3.5-Omni (Apr 2026): hybrid-attention MoE Thinker and Talker, 256k context, and ARIA, which dynamically aligns text and speech units to stabilise streaming speech and prosody (arXiv 2604.15804). The authors claim SOTA on 215 audio and audio-visual subtasks.
• Kimi-Audio: continuous audio features in, discrete tokens out, more than 13M h of audio pretraining (arXiv 2504.18425).
• MiniCPM-o 2.6: 8B, built from SigLip-400M, Whisper-medium-300M, ChatTTS-200M and Qwen2.5-7B, with a time-division multiplexing mechanism for streaming (OpenBMB model card).
• Commercial: OpenAI's Realtime API model "works directly with audio" (OpenAI docs). Google's Live API offers native audio output models (Gemini API docs).

The intelligence gap

Speaking costs reasoning quality. Causes reported: catastrophic forgetting and modality inequivalence (arXiv 2509.14930), and a bottleneck in condensing redundant speech into stable late-layer decisions (arXiv 2603.01502). Mitigations: a frozen LLM (arXiv 2411.00774), cross-modal distillation from a text teacher (arXiv 2509.14930), layer-wise hidden-state alignment with RL (arXiv 2601.05543), EchoX (arXiv 2509.09174).

Full-duplex and backchannelling

Full duplex means the model listens and speaks at once, so it can overlap, interrupt and backchannel without explicit turn logic. This is the target capability for the Synthesia voice role.

Schemes

Reported Self-reported results. The DuplexSLA authors report 98.33% backchannel accuracy on their own benchmark, against 40% or less for the closed-source baselines they tested (arXiv 2605.20755). NVIDIA's PersonaPlex model card lists a smooth turn-taking rate of 0.908 and latencies of 0.170 and 0.240 on Full-Duplex-Bench, without units on the card (model card; paper arXiv 2602.06053).

Benchmarks

• Full-Duplex-Bench v1 (arXiv 2503.04721): pause handling, backchanneling, turn-taking, interruption management; automatic metrics.
• v1.5 (arXiv 2507.23159): overlap scenarios (user interruption, user backchannel, talking to others, background speech), stop and response latency. It finds two strategies: responsive and floor-holding.
• ECHO (arXiv 2609.17360): paired Chinese yield-or-keep turn-taking benchmark; the authors find three of four systems over-yield on backchannels.
• VoiceBench (arXiv 2410.17196): multi-faceted evaluation of LLM-based voice assistants.
• URO-Bench (arXiv 2502.17810): open-source spoken dialogue models lag behind their backbone LLMs on instruction following.

Evaluation is maturing but not comparable across papers.

Data is the bottleneck

A 2026 survey (arXiv 2606.19453) proposes an L0-L3 hierarchy and an IDLE/LISTEN/SPEAK/WAIT/DUAL state machine. It finds that observed behaviour is constrained by the interaction patterns in training and evaluation, and that public training-data coverage is limited. A 2025 survey (arXiv 2509.14515) names synchronous data scarcity, architectural divergence and evaluation gaps as the main challenges.

Multimodal, multilingual S2S

Multilingual S2S is lopsided: models understand far more languages than they speak. Text is the bridge that makes the rest possible.

Coverage asymmetry

Qwen3-Omni (arXiv 2509.17765) and Qwen3.5-Omni (arXiv 2604.15804) each report speech generation in 10 languages. Inference Generation is likely the constraint, because it needs clean TTS-grade data per language.

Translation and code-switching

• Hibiki (arXiv 2502.03382): simultaneous speech-to-speech translation with multistream modelling. It uses per-word delays from text machine translation to build aligned synthetic data. The authors' model has 2.7B parameters and translates French to English (the pair they evaluate).
• SeamlessStreaming (arXiv 2312.05187): direct streaming translation from 101 source languages into speech in 36 target languages.

How text streams serve S2S

Rates, frames and clocks

A speech system runs several clocks at once, and "the frame rate" can mean any of them. Keep the clocks apart and every number in this chapter becomes easy to check. The anchor to remember is 12.5 Hz, which is one frame every 80 ms.

The ladder from waveform to LLM

The whole chain for one Whisper-style input path:

Other encoders reach 12.5 Hz by other routes. Step-Audio 2 runs a frozen audio encoder at 25 Hz and a 2x adaptor, giving 12.5 Hz (arXiv 2507.16632).

The reason to stop at 12.5 Hz is sequence length. Thirty minutes is 1,800 s, so 50 Hz gives $1800 \times 50 = 90{,}000$ positions (derived). At 12.5 Hz the same audio is $1800 \times 12.5 = 22{,}500$ positions (derived).

Codec frame rates, tokens per second, bitrate

A codec frame is a vector in time. A token is one integer id. A residual quantiser (RVQ) turns each frame into one id per codebook, so token rate and frame rate differ.

Mimi has $f = 12.5$ Hz, $N = 8$ and $V = 2048 = 2^{11}$, so $12.5 \times 8 = 100$ tokens/s and $100 \times 11 = 1100$ bps. That matches the 1.1 kbps in the Mimi paper (arXiv 2410.00037). Confirmed

Text rate, audio rate and duplex clocks

Text is slow and audio is fast, so a model that emits both must choose how to align them. This section shows the two alignments and the clock a duplex model adds on top.

Text tokens against audio frames

Text is slow. Moshi puts text on the same 12.5 Hz grid as audio and fills the empty frames with a PAD token. In English conversational speech padding is about 65% of text-stream tokens (arXiv 2410.00037), so about 35% of frames carry a real token, roughly $0.35 \times 12.5 \approx 4.4$ text tokens/s (derived).

Interleaving models drop the shared grid and fix an order instead. GLM-4-Voice (arXiv 2412.02612) alternates 13 text tokens with 26 speech tokens (1:2). At 12.5 Hz, 26 speech tokens last $26 / 12.5 = 2.08$ s, so the text runs at $13 / 2.08 = 6.25$ tokens/s (derived).

Duplex clocks

A duplex model needs one clock that both directions share. In Moshi, every 80 ms step reads the user's 8 audio codes and emits the agent's text token and 8 audio codes. There is no turn state; silence is just tokens. One temporal-transformer pass plus up to 17 depth sub-steps must finish before the next frame is due. Inference If the temporal pass were free, the budget is $80 / 17 \approx 4.7$ ms per sub-step (derived).

Other systems decide less often than they run:

Latency in frames, video alignment and vocabulary

Latency is easiest to reason about in frames. The same clock arithmetic also aligns audio with video, and fixes the vocabulary for the interview.

Seconds to frames, and Moshi's latency

Divide any delay by 0.08 s to count frames.

Moshi's 160 ms is one 80 ms frame plus one 80 ms acoustic-delay step: $2 \times 80 = 160$ ms (derived decomposition). The paper reports about 200 ms in practice. With acoustic delay 2 the paper states a theoretical latency of 8 timesteps, 640 ms (arXiv 2410.00037). Confirmed

Video alignment

Video frames run at 25 or 30 fps, against 12.5 Hz for audio tokens. At 25 fps there are exactly 2 video frames per audio frame. At 30 fps there are $30 / 12.5 = 2.4$, which is not an integer, so the two streams cannot share a frame grid (derived).

Video latents are coarser than pixel frames. Wan 2.1's causal VAE has a latent length of about $1 + T/4$ for $T$ frames (arXiv 2503.20314), so 25 fps becomes 6.25 latent frames per second and 30 fps becomes 7.5 (derived).

Ovi aligns the two token rates without resampling. It runs RoPE on both streams and scales the audio RoPE frequencies by $31/157 \approx 0.197$, so 31 video latent frames line up with 157 audio tokens (arXiv 2510.01284). That is about $157 / 31 \approx 5.06$ audio tokens per video latent frame (derived).

Say it right in the interview

Timing logic: choosing input and output rates

Two rates matter when you design a speech model: how fast audio enters the LLM, and how fast the LLM must emit audio frames. They are set by different forces, and the output rate is the one that fixes your latency floor. The clocks themselves are defined in Rates, frames and clocks; this section covers how to choose them. Numbers are cited to the paper; arithmetic is marked derived, reasoning is marked Inference.

The input path, step by step

1. 16 kHz waveform, cut into 25 ms windows with a 10 ms hop: 100 Hz mel frames (arXiv 2212.04356).
2. A Whisper-style encoder has a stride-2 convolution: 50 Hz, 20 ms per frame (arXiv 2212.04356; 50 Hz is derived).
3. A pooling or adapter stage cuts the rate again, to 10 to 25 Hz before the LLM.

Inference Input can be fast and continuous because the audio already exists. The encoder reads it in parallel with no sampling loop, so a high rate only costs encoder compute. Output must be generated one step at a time, each step conditioned on the last, so every extra frame per second is an extra serial LLM step. The pooling stage exists to move that cost out of the LLM, where each position is expensive.

The output path: what fixes the rate

The output rate is the frame rate of the audio codec (or tokenizer) the LLM predicts. Each frame becomes one id per codebook, so $\text{tokens/s} = f \times Q$, and the bitrate adds the bits per id:

Mimi: $12.5 \times 8 \times 11 = 1100$ bps (derived), matching the 1.1 kbps in the paper (arXiv 2410.00037). The paper states frame and stride correspond to 80 ms, and that 8 codebooks at 12.5 Hz give "a sequence length of 100 steps per second", against around 3 to 4 text tokens per second for English.

The deeper table of codec bitrates is in Codec frame rates, tokens per second, bitrate.

How to choose the output rate

• Sequence length and compute. Mimi's 100 steps per second is the figure the paper weighs against 3 to 4 text tokens per second (arXiv 2410.00037). VibeVoice's 7.5 Hz is chosen to cut decoding steps (arXiv 2508.19205).
• Reconstruction quality. WavTokenizer scores PESQ 1.70 and UTMOS 3.60 at 40 tokens/s against 2.37 and 4.05 at 75 (arXiv 2408.16532, Table 1). Pooling X-Codec 2 to a 25 Hz variant raises UTMOSv2 on seven languages, for example Dutch 2.168 to 2.457 (arXiv 2601.20185).
• Intelligibility and LM difficulty. SiTok WER is 23.0 at 6.25 Hz, 4.06 at 12.5 Hz and 3.05 at 25 Hz (arXiv 2602.06602). TaDiCodec reaches 6.25 Hz with WER 3.02, but removing the text input at decode causes "significant performance degradation under extremely low token rate" (arXiv 2508.16790). A probing study measures WavTokenizer WER at 11.37% (75 tokens/s) against 24.36% (40 tokens/s) (arXiv 2606.16969, Table 1); it says models below 50 Hz without architectural accommodations degrade, and it attributes its own DAC cliff to a training misconfiguration, so it is not evidence of an intrinsic floor.
• Text-rate matching. GLM-4-Voice alternates 13 text tokens and 26 speech tokens, a 1:2 ratio chosen so text generation is "consistently faster than speech" (arXiv 2412.02612). Details: text and audio clocks.
• Reaction granularity. Inference One frame is the smallest unit the model can emit or react to: 80 ms at 12.5 Hz, 40 ms at 25 Hz.

Why 12.5 Hz

Three pressures meet at 12.5 Hz, and a fourth sets the cost.

• Sequence length and compute. Voxtral cuts its 50 Hz encoder output with a 4x adapter to 12.5 Hz (arXiv 2507.13264). Thirty minutes (1,800 s) at 50 Hz is $1800 \times 50 = 90{,}000$ positions; at 12.5 Hz it is 22,500 (derived).
• Text-rate matching. Moshi puts English at around 3 to 4 text tokens per second against 100 audio steps per second at 8 codebooks (arXiv 2410.00037). At 12.5 Hz that is about $12.5 / 4 \approx 3$ frames per text token (derived).
• The quality knee. SiTok WER is 23.0, 4.06 and 3.05 at 6.25, 12.5 and 25 Hz (arXiv 2602.06602). Going from 12.5 to 25 Hz buys little; going to 6.25 Hz costs a lot.
• Reaction quantum. An 80 ms frame is shorter than the median 151 ms by which human listeners start speaking before the other turn ends in smooth floor transfers (arXiv 2608.25218). Inference So the frame is not the binding constraint on human-like timing.
• The cost. Pooling discards fine acoustic detail. Kimi-Audio keeps continuous 50 Hz Whisper features alongside its 12.5 Hz discrete semantic tokens (arXiv 2504.18425).

Who emits what, per frame

The LLM does not predict one text token per input speech frame. How text lines up with frames depends on the design:

So text and speech are 1:1 per frame only in time-aligned designs, and that is why those designs pad the text stream.

Decoupling the backbone rate from the output rate

If every codebook needed its own backbone step, Q codebooks at 12.5 Hz would force a $12.5 \times Q$ Hz backbone. Designs avoid this by letting one backbone step cover many tokens:

• Moshi: a depth transformer fills K = 2Q + 1 = 17 sub-tokens within each 80 ms frame (arXiv 2410.00037).
• Qwen3-Omni Talker: a linear head predicts codebook 0 per frame, and an MTP module generates the residual codebooks (arXiv 2509.17765).
• VITA-Audio: ten audio tokens in a single forward pass, 3 to 5x speed-up (arXiv 2505.03739).
• MusicGen: a delay pattern shifts codebook k by k-1 steps, so one pass predicts all codebooks (arXiv 2306.05284).
• Patches: VibeVoice and DiTAR predict a patch per step, see continuous autoregression.
• DiffuPlex predicts multiple future user and assistant frames in one backbone wake: 1.46x and 1.59x wall-clock speedups (arXiv 2610.12214).

Minimums and rules of thumb

1. Pool the encoder to 12.5 to 25 Hz before the LLM (arXiv 2507.13264, 2509.17765, 2412.02612, 2504.18425).
2. Default output rate 12.5 Hz (arXiv 2410.00037, 2412.02612, 2509.08753, 2602.06602); SiTok shows a steep WER penalty below it.
3. Go to 25 Hz when reconstruction is the bottleneck (arXiv 2412.10117, 2601.20185; SiTok has its best WER at 25 Hz, arXiv 2602.06602).
4. A lower frame rate is cheaper for the LM (arXiv 2306.06546, 2410.00037).
5. When interleaving, let text lead speech (arXiv 2412.02612 at 1:2; Moshi's 3 to 4 text tokens/s, arXiv 2410.00037).
6. With many codebooks, emit several tokens per backbone step (arXiv 2410.00037, 2509.17765, 2505.03739, 2610.12214).
7. Inference The minimum reaction time is one frame. Practical chunks are larger: Moshi's latency is 160 ms in theory and 200 ms in practice (arXiv 2410.00037); SyncLLM uses chunks of 160 to 240 ms (arXiv 2409.15594).
8. Inference from the source 6.25 Hz is viable only when the transcript is available at decode time (arXiv 2508.16790, ablation).

Worked example

Derived from the rates above, not measured.

The 25 Hz design reacts twice as finely but runs the backbone twice as often (derived). The 12.5 Hz design runs the backbone half as often but adds a depth pass inside each 80 ms deadline. See the per-frame deadline.

Producing text and speech together: Moshi

Voice-to-voice models that "reason and talk" must emit text and audio at the same time. The sources show three coupling patterns. This section covers Moshi, the delayed parallel stream; the next covers delay as a knob and fixed interleaving. The control head that decides when to speak is in the full-duplex sections.

Moshi: the codec and the clock

Moshi (arXiv 2410.00037) models speech with Mimi tokens at 12.5 Hz. Mimi has 8 RVQ levels of 2048 codes. The first level is a semantic code distilled from WavLM; the other seven are acoustic. The codec is causal, with an 80 ms frame. Confirmed

Each 80 ms frame holds 17 streams: 1 text token, 8 Moshi audio tokens and 8 user audio tokens ($K = 2Q + 1 = 17$ with $Q = 8$). The user stream is just another input: the model hears the user continuously while it speaks.

Two transformers per frame

• Temporal transformer: 4096-dim, 32 layers, run once per frame over time. It is initialised from the Helium text LM (about 2.1T training tokens) and outputs a context vector.
• Depth transformer: 1024-dim, 6 layers, random initialisation. It turns the context vector into up to 17 sub-tokens inside the frame.
• The split keeps the big model at 12.5 steps per second. Only the small model runs 17 times per step. Inference

Inner Monologue: text, PAD and delays

Moshi predicts a text token for the agent's own speech on the same 12.5 Hz grid. Frames with no new word carry PAD. The paper also defines an EPAD token, inserted before the next word to mark the end of the padding. Padding is about 65% of text-stream tokens in English conversational speech.

Text is the first sub-step of each frame, so that frame's audio tokens are conditioned on it (derived from the sub-step order: text, Moshi audio, user audio). Inference

Latency follows from the delay: 160 ms theoretical with acoustic delay 1, about 200 ms in practice. The paper also gives 640 ms for acoustic delay 2. The recipe: Helium text LM, then 7M h of audio transcribed with Whisper large-v3 (single stream), then multi-stream post-training on diarised audio, a Fisher fine-tune (2,000 h of phone conversation), and instruction tuning on more than 20k h of synthetic speech (arXiv 2410.00037). Reported

Ways to model streaming ASR and TTS in speech LLMs

Streaming ASR and streaming TTS each split into a handful of families, and each family answers two questions. For ASR: what sets the latency floor, and who decides when to emit a word? For TTS: does the text arrive complete or incrementally, how far does the decoder look ahead, and how many tokens come out per step? Numbers are the authors' own, on different hardware and test sets, so compare shapes, not digits. Background is in Streaming approaches for ASR, Streaming and low latency, Moshi and Delay and interleaving.

Streaming ASR families

Chunked (SpeechLLM-XL). A decoder-only model reads audio in fixed chunks and emits the chunk's text followed by an end-of-chunk token. Training uses CTC forced alignment and an auxiliary CTC loss weighted 0.5. Chunk length is the latency dial: test clean / other WER is 2.5 / 6.5 at 2.56 s, 2.7 / 6.7 at 1.28 s and 3.1 / 7.8 at 0.32 s (arXiv 2410.03752).

Wait token (Speech ReaLLM). An RNN-T-style blank or wait token is predicted after every input embedding, so the LLM decides per frame whether to emit. Latency is the 240 ms input frame plus 960 ms of encoder right context. The 80M model reaches WER 3.0 / 7.4 on LibriSpeech, with RTF 0.79 under greedy decoding (arXiv 2406.09569).

Transducer with an LLM predictor (Transducer-Llama). The transducer keeps its streaming alignment and uses a Llama as the prediction network. With 160 ms segments and a Llama3-8b predictor, WER is 2.47 / 6.53 against 3.65 / 8.96 for a standard RNN-T. The paper uses a weak-to-strong LM swap and MWER fine-tuning (arXiv 2412.16464).

Delayed stream (Kyutai DSM). Text and audio share a clock and the text is simply delayed, 2.5 s by default for ASR. DSM-ASR scores 1.7 / 4.3 on LibriSpeech clean / other, against 3.3 / 4.7 for streaming Parakeet-TDT-v2, with delay precision of about 300 ms (arXiv 2509.08753). This is the family from Delay and interleaving.

Read/write policies. These wrap an offline model. LocalAgreement takes the longest common prefix of n = 2 successive updates over a buffer of about 30 s trimmed at sentence boundaries, at 3.3 s latency on long-form audio in the abstract (Table 3: 3.62 s at a 1.0 s minimum chunk; arXiv 2307.14743). Simul-Whisper stops decoding at the end of the received audio using cross-attention, with truncation detection (arXiv 2406.10052). Its abstract reports 1.46% average degradation at a 1 s chunk; Table 1 gives 2.34 for Large-v2, so I treat the abstract figure with care.

Unified streaming and offline (Qwen3-ASR). The AuT encoder runs at 12.5 Hz with a flash-attention window of 1 to 8 s, so one model serves both modes. The 1.7B model averages 2.69 offline and 3.33 streaming (2 s chunks in the evaluation); the 0.6B model reaches a time to first token as low as 92 ms with vLLM (arXiv 2601.21337).

Inference Two families are not covered because I found no primary source: CTC partials with LLM correction, and learned read/write agents.

Streaming TTS families

1. Full text, causal decoder (Qwen3-TTS). The tokenizer runs at 12.5 Hz and is causal with no look-ahead, so a packet of 4 tokens covers 320 ms. For the 0.6B model, time to first packet is 93 ms for the LM plus 4 ms of decode, 97 ms in total; the 1.7B model takes 101 ms (Table 2; arXiv 2601.15621). The Sesame CSM README documents a single generate call that returns a complete clip and no streaming mode (github.com/SesameAILabs/csm).

2. Incremental text, interleaved. CosyVoice 2 streaming mixes text and speech tokens at N = 5 to M = 15 (arXiv 2412.10117). SyncSpeech starts speaking after 2 text tokens, where CosyVoice 2 needs 5; its first packet takes 0.06 s against 0.22 s for CosyVoice 2 on LibriSpeech test-clean (arXiv 2502.11094). SpeakStream emits each word's speech after the word, using forced-aligned boundaries, at about 30 ms TTS latency (arXiv 2505.19206). This family fits a TTS that consumes another LLM's streamed output.

3. AR LM plus chunk-wise causal flow. CosyVoice 2 trains one flow-matching model under four masks: full-causal, chunk-M, chunk-2M and non-causal. Each mask sets how much future context a chunk sees (arXiv 2412.10117). The paper gives latency formulas, not a measured first packet. Kimi-Audio cuts audio into chunks of about 1 s and appends the next n semantic tokens (for example 4) from the following chunk before decoding. This is training-free and delays the first chunk by n tokens (arXiv 2504.18425, section 2.4).

4. Multi-token prediction (VITA-Audio). The MCTP module predicts about 10 audio tokens per forward pass. The audio token delay is 0, against 1 for Moshi and 13 for GLM-4-Voice (Table 1). The streaming first chunk drops from 236 ms to 53 ms (Table 9; arXiv 2505.03739).

5. Fixed ratio in dialogue models. GLM-4-Voice interleaves 13 text : 26 speech tokens in 0.8 s blocks (arXiv 2412.02612). MinMo uses 5 semantic : 15 speech tokens, with about 600 ms end to end (arXiv 2501.06282). OmniFlatten uses 2 : 10 and reports 193 ms against Moshi at 553 ms (arXiv 2410.17799). LLaMA-Omni 2 reads R = 3 and writes W = 10 tokens; its first speech chunk takes about 583 ms on 7B, the sum of LLM, TTS, flow and vocoder time (arXiv 2505.02625).

Inference A sixth family, non-autoregressive or flow models run chunk by chunk, is not covered because I found no primary streaming paper for F5 or ZipVoice.

How to choose

Inference This maps onto the unified speech LLM design. The listen head is a delayed-stream or chunked ASR. The speak head is incremental-text interleaved TTS, consuming the tokens that the downstream reasoning LLM streams out.

Delay as a knob and interleaved decoders

Moshi's text-audio offset is a design choice, and other systems make the same choice differently. This section covers delay as a knob (DSM, Hibiki) and the fixed-ratio interleaving families.

DSM: delay as a knob

Delayed Streams Modeling (arXiv 2509.08753) pre-aligns streams and applies a fixed delay per stream. In the paper's framing, ASR corresponds to the text stream being delayed relative to audio, and TTS to the opposite.

Inference Delay becomes a per-stream setting, not an architecture. ASR needs audio lookahead before it commits a word. TTS needs text lookahead before it commits prosody. For "text and voice at once", the same decoder with a chosen offset is the lever.

Hibiki: per-word delay

Hibiki (arXiv 2502.03382) is a multistream decoder-only model for speech translation. It sets a delay per word, using the perplexity of an off-the-shelf text translation model (MADLAD-3B) on source prefixes. The target is required to lag the source by at least 2 s. The authors report LAAL of 3.4 s on short-form and 5.0 s on long-form audio, against Seamless at 2.8 s and 4.2 s. On the same tables Hibiki scores ASR-BLEU 35.5 against Seamless at 33.8 (short-form) and 26.6 against 23.9 (long-form).

Interleaving and streaming decoders

Reported All latencies are the authors' own figures under different hardware and definitions.

Inference Moshi aligns text to the clock (padding fills the gaps). Interleaving aligns text to the order (no clock, so no padding, but no free time axis). Thinker-Talker separates the two jobs: the Thinker can finish a thought before the Talker voices it. I found no source here that shows sustained reasoning and live speech together in one duplex model.

Full-duplex models compared

A full-duplex model listens while it speaks. Backchannels, overlaps and barge-in then come from the model's own decisions, not from a VAD timer. The sources group the designs by where the "speak or listen" decision lives.

Survey levels L0 to L3

A 2026 survey (arXiv 2606.19453) defines four levels. L0 is an external scheduler around a half-duplex model. L1 is a control head reading the LLM's hidden states. L2 encodes the duties in the token stream itself. L3 is representation-level and not realised yet. Its line to remember: "architecture sets capacity, training realizes it".

Inference My mapping: control-head models are L1; dual-stream, flattened, chunked and action-channel models are L2.

Families and numbers

Reported Most figures are authors' own. On DuplexSLA-Bench (self-reported, arXiv 2605.20755), backchannel accuracy is 0.33% for gpt-realtime-1.5 (semantic-vad-high), 40.00% for gemini-3.1-flash-live and 98.33% for DuplexSLA. Instruct-FD (arXiv 2607.20460) finds the best of six full-duplex systems follows turn-taking instructions 64.4% of the time.

Wiring the decision and the turn-taking feel

The same families can be read as three wirings of one decision. The wiring explains most of what makes a duplex model feel like, or unlike, a person talking.

Three ways to wire the decision

• Dual-stream: no separate decision; speaking and listening are both tokens on every frame.
• Flattened: one sequence, so who speaks is decided by which role's tokens come next.
• Control head: a small classifier reads hidden states and gates a mostly unchanged LLM. This is why Freeze-Omni can keep its LLM frozen.

The 2026 survey (arXiv 2606.19453) calls SyncLLM an L2-shaped but "apparent (not substantive)" full-duplex system.

What causes the turn-taking feel

Inference Two causes follow from the sources, both about the decision and not about speech quality:

1. Decisions quantised at chunk boundaries. A decision waits for the next boundary, $c/2$ on average for chunk length $c$ (derived). That is 40 ms for an 80 ms frame, 120 ms for a 240 ms chunk and 500 ms for a 1.0 s chunk.
2. Scarce interaction data. The 2025 survey names synchronous data scarcity (arXiv 2509.14515). The 2026 survey finds that observed behaviour is constrained by the interaction patterns in training and evaluation (arXiv 2606.19453).

A related finding: audio foundation models miss when to speak, interrupt too aggressively and rarely backchannel (Talking Turns, arXiv 2503.01174). Inference My reading: the speech path is not the bottleneck. The bottleneck is the duplex decision, plus ingesting user audio while the model is speaking its own.

Turn-taking prediction and the human baseline

Turn-taking is a prediction problem: when does the other speaker stop, and should I start? The field moved from stand-alone predictors to end-of-turn detectors in cascades, and now to duplex models that fold the decision into generation.

Predictors: TurnGPT and VAP

• TurnGPT (Ekstedt and Skantze, Findings of EMNLP 2020, arXiv 2010.10874): a transformer language model that predicts turn shifts in spoken dialogue from text.
• Voice Activity Projection, VAP (Ekstedt and Skantze, Interspeech 2022, arXiv 2205.09812): a self-supervised predictive objective on the voice activity of both speakers, so turn-taking models train without labels.
• Multilingual VAP (LREC-COLING 2024, arXiv 2403.06487): English, Mandarin and Japanese. A model trained on one language predicts poorly on the others; one multilingual model performs on par with the monolingual ones and learns to identify the input language.

End-of-turn models in production

Reported These are vendor numbers, measured on different benchmarks and conditions, so do not rank the vendors from this table. The eot-bench framing is worth borrowing: false-cutoff rate at a fixed latency budget (300 or 600 ms), and mean latency at a fixed false-cutoff target (5% or 10%). It compares systems at an operating point, not at a single threshold.

The human baseline

On smooth transfers, humans start speaking a median of 151 ms before the other turn ends (TurnBench, arXiv 2608.25218). Define the floor-transfer offset as

A cascade that waits for silence can only produce $\Delta > 0$. A negative offset needs prediction, which is what VAP and duplex models do. Inference

TurnBench (arXiv 2608.25218, Aug 2026) is an annotated benchmark of end-of-turn (EOT) and interruption detection. The authors report about 30 h of dual-channel dyadic speech in 154 dialogues, each triple-annotated. A segment is gold when at least 2 of 3 annotators give the same label with endpoints within $\pm 200$ ms. Results are recall, false-positive rate (FPR) and median latency, across 14 turn-taking systems.

The TurnBench authors state the open problem: no current system matches human timing without excessive false positives.

Backchannels, barge-in and self-voice

A good duplex agent tells a backchannel from a barge-in, and does not mistake its own voice for the user's. Both are decision problems with an echo problem underneath.

Backchannel against barge-in

A backchannel ("mm-hm") signals attention and leaves the floor with the speaker. A barge-in takes the floor. Inference A characteristic failure follows: the system treats a backchannel as a barge-in and stops talking.

• Reported DuplexSLA-Bench (arXiv 2605.20755, Table 6, context-prefill setting, authors' numbers): backchannel accuracy 0.33% for gpt-realtime-1.5 with semantic VAD, 40.00% for gemini-3.1-flash-live, 98.33% for DuplexSLA.
• Talking Turns (ICLR 2025, arXiv 2503.01174): in a user study, audio foundation models sometimes miss when to speak, interrupt too aggressively and rarely backchannel. It uses a supervised turn-event model as judge, with test benchmarks curated from Switchboard.
• Full-Duplex-Bench v1.5 (arXiv 2507.23159) scores user interruption, user backchannel, talking to others and background speech. It measures categorical behaviour, stop and response latency, and prosodic adaptation.

Inference Two numbers define good barge-in handling: stop latency, the time from user onset to the last audible agent sample, and the false-yield rate, how often a backchannel, laugh or background voice stops the agent. Optimising only one of them breaks the other.

The full metric proposals, with operating points, are in conversational evaluation; training for these behaviours is in online RL and rewards.

Self-voice and echo

A duplex model hears the speaker output it just produced, unless something removes it. MinMo's duplex module still requires separate acoustic echo cancellation (AEC) and VAD modules (arXiv 2501.06282). SALMONN-omni (arXiv 2411.18138) is instead described as listening to its own generated speech and background sounds while speaking, and SALM-Duplex (arXiv 2505.15670) targets barge-in and turn-taking.

Inference A model that hears its own output needs an explicit reference-signal path: the playback signal goes in as an extra input, so the model or the AEC can subtract what it said from what the microphone heard. Measure stop latency on echo-cancelled output, or a barge-in test passes for the wrong reason.

Streaming inference: deadlines and first packets

A duplex model is a real-time loop, not a request-response call. It must produce a frame on a fixed clock for as long as the session lasts, even when nobody is talking. This section turns the earlier rates into engineering budgets.

The per-frame deadline

Inference Moshi (arXiv 2410.00037) runs at 12.5 Hz, so it has a hard deadline of 80 ms per frame: one temporal pass plus up to 17 depth sub-steps (the paper's K = 17 streams). A missed deadline is an audible gap, because the audio must keep flowing.

The standard check is the real-time factor (RTF), compute time divided by audio time:

RTF below 1 is necessary and not sufficient: the worst frame must still fit in 80 ms. MiniCPM-o 4.5 reports RTF 0.21 (INT4) to 0.27 (FP16) with llama.cpp-omni on an RTX 4090 (arXiv 2604.27393). Reported That is 210 to 270 ms of compute per second of audio (derived).

LLaMA-Omni 2 (arXiv 2505.02625) shows the chunk version of the same arithmetic. Speech tokens run at 25 Hz, so each is 40 ms and a 10-token chunk is 400 ms. The authors report a 542.7 to 663.3 ms first chunk, so roughly 140 to 260 ms of overhead sits on top of the chunk (derived). Inference

The deadline can be relaxed by waking the backbone less often than once per frame. DiffuPlex (arXiv 2610.12214) predicts several future user and assistant frames in one backbone wake, using rolling masked diffusion over an 80 ms interaction interval. The authors report 1.46× (LISTEN) and 1.59× (SPEAK) deployment wall-clock speedups, and 1.61× and 1.80× for the core LM alone.

Chunk size trade-offs

MiniCPM-o's ablation (arXiv 2604.27393) finds 1.0 s gives the best balance. The authors report that shorter chunks leave less modelling budget for control and generation, and that too-short chunks lack enough information per window, causing substantial degradation. Inference Shorter chunks cut the average wait for a decision ($c/2$, derived) but pay the per-step cost more often. Smaller is not automatically better.

First packet against chunk

The first packet is the time until the first playable audio. The chunk is the granularity after that. They are different numbers.

Streaming inference: memory and many sessions

Memory, not compute, often sets how many duplex sessions one GPU can hold. This section covers the cache, long sessions and batching.

KV-cache growth

Inference This arithmetic is derived under stated assumptions: full multi-head attention, bf16 storage, and only the temporal transformer counted (32 layers, model dimension 4096, from Moshi's Table 1, arXiv 2410.00037). The depth transformer's cache can be discarded each frame.

The leading 2 is keys plus values; the trailing 2 is bytes per bf16 value. At 12.5 frames per second that is 512 KiB $\times 12.5 \approx 6.3$ MiB per second, or about 23 GB for one hour (derived). Grouped-query attention would shrink it.

The cache grows linearly with session length and with the number of sessions. Inference For many concurrent sessions, cache memory, not weights, sets capacity.

Long sessions

• The Moshi paper states the model follows multi-turn conversations across 5 minutes (arXiv 2410.00037). Ask what happens beyond that before claiming more.
• One windowing pattern for audio is block-local attention in Qwen2.5-Omni's streaming codec DiT: 2 blocks of lookback and 1 of lookahead (arXiv 2503.20215).
• For video, Self Forcing (arXiv 2506.08009) uses a rolling KV cache for autoregressive video extrapolation at inference. It is a related pattern for unbounded streams, but it is a video model.

Batching many duplex sessions

Inference A turn-based server pays nothing for an idle chat. A duplex session is always active: it needs one frame every 80 ms, including silence. Batching sessions into one GPU step helps throughput, but the whole batched step still has to finish inside 80 ms. The sustainable batch size is the largest $B$ with $t_{\text{step}}(B) < 80$ ms.

DSM-TTS reports 150 ms latency at batch size 1 and about 380 ms at batch size 32 (arXiv 2509.08753, Table 6), so latency figures depend on load.

Streaming decoders and the latency budget

The decoder streams too, and its lookahead is latency. This section ends with the end-to-end budget and a checklist.

Streaming codec decoding

The decoder must also stream, and any lookahead it needs is added latency.

Inference One block of lookahead costs one block of delay. A fully causal decoder such as Mimi's pays none, which is part of why 160 ms is achievable.

Where latency hides

Checklist

• Report latency with its definition: theory, first packet, first chunk, turn-taking delay or total.
• State batch size, GPU and whether the run was cold start.
• Measure the worst frame, not just RTF; a missed 80 ms deadline is audible.
• Compute the KV-cache cost per session before promising concurrency.
• Test barge-in on echo-cancelled output and report stop latency.
• Add the decision wait $c/2$ when comparing chunked and per-frame systems.
• Run a session past the reported context (Moshi: 5 minutes) to see what happens at the limit.

Pre-training and post-training speech-to-speech models

Nearly every open speech-to-speech model follows one recipe. Start from a strong text LLM, add speech tokens, pretrain on ASR, TTS and interleaved speech-text, run spoken-dialogue SFT, then apply DPO or RL on the text side and sometimes the speech side. The real choice is whether the text backbone stays frozen or trains on a mixed corpus.

Every recipe below is the authors' own account in the cited paper. For the TTS-only post-training view (WER and SIM rewards, hacking), see Post-training and evaluation for speech generation. The reward details for online RL are in the next section.

Stage-by-stage recipes

Reported All rows are the authors' own accounts.

Frozen backbone or mixed text: fighting forgetting

Confirmed Adding speech capability can degrade knowledge and reasoning even for text inputs, and more for spoken queries (arXiv 2509.14930). SALAD (arXiv 2510.13632) reports that speech-adapted LLMs underperform their text counterparts and attributes the gap to forgetting and cross-modal misalignment.

Interleaved synthetic pretraining and spoken instruction data

Interleaved pretraining (arXiv 2411.17607) synthesises interleaved speech tokens from text spans with a text-to-token model and a 12.5 Hz supervised tokenizer. It used 1T tokens including 600B synthetic interleaved. Spoken QA rose from 13% (Moshi) to 31%.

Spoken instruction data is synthetic almost everywhere: an LLM writes the dialogue and TTS renders it. Moshi uses over 20k h of synthetic speech (arXiv 2410.00037), and Step-Audio 2 adds synthesised speech-to-speech data in its cooldown (arXiv 2507.16632). Inference The risk is TTS voice and prosody leakage into the model's own speaking style; mix in real recorded turns and many TTS voices.

Preference, RL and distillation

• DPO on text tokens. Moshi-Aligned (arXiv 2506.21463) applies length-normalised DPO to text-token probabilities only, because audio-token probabilities are unstable. It uses 93,490 pairs (27% timing, 73% content), judged by Mistral Large 2. Averaged QA moved 36.1 to 39.2 and safety 70.9 to 77.8 (Moshi-Instruct to Moshi-Aligned).
• Talker DPO. Qwen2.5-Omni ranks preference triplets by WER and punctuation-pause error. Qwen3-Omni runs DPO on multilingual pairs in a four-stage Talker.
• RL. Step-Audio 2 runs PPO with a binary conciseness reward on the thinking trace (60 iterations), PPO with a learned reward model (120), then GRPO for audio perception (400). Qwen3-Omni uses GSPO with rule-based and model-based rewards.
• Distillation. Qwen3-Omni distils strong to weak: off-policy, then on-policy KL against Qwen3-32B and 235B teachers.

Duplex post-training and evaluation gates

DuplexPO (arXiv 2607.07148) runs GRPO-style RL on selected windows from long human conversations. Its four-part reward (FCDR) covers turn initiation, backchanneling, yielding and regularised participation, with a Qwen2.5-7B-Instruct backbone. Reported Intelligence gains are small (Llama Questions 75.3 versus 72.0 for the SFT baseline, Table 4). The point is that timing is optimised without touching content.

Synthesis

Proposed recipe for a full-duplex text+voice model

Online RL and rewards for speech generation

The common pattern: the policy is a codec or discrete-token LM (sometimes flow matching), sampled in groups. Rewards come from the decoded audio: ASR error for intelligibility, speaker-verification (SV) cosine for similarity, a quality or preference predictor for naturalness. The main failure is reward hacking on any single optimised score. Duplex timing is the newest axis, with per-axis rewards derived from VAD events.

The TTS-side summary (DiffRO, F5R-TTS, hacking signatures, listening gates) is in Post-training and evaluation for speech generation. This section adds reward implementations, the duplex rewards, and infrastructure.

Reward taxonomy and concrete implementations

Duplex timing rewards

• DuplexPO (arXiv 2607.07148). The Factorized Conversational Dynamics Reward covers turn initiation, backchanneling, yielding and regularised participation. It optimises only windows around annotated agent events, GRPO-style.
• Kyutai multi-faceted alignment (arXiv 2606.11167). Reported RL post-training of Moshi and PersonaPlex with one reward per axis (pause handling, turn-taking, backchanneling, user interruption) plus an LLM-based response-quality reward against semantic degradation. Per the official model card, GRPO uses rule-based rewards, segments come from Meta's Seamless Interaction corpus via VAD, and evaluation uses Full-Duplex-Bench v1 and v2. Checkpoints: kyutai/moshika-rl-seamless and kyutai/personaplex-rl-seamless.
• ASPIRin (arXiv 2604.10065). GRPO with a binary speaking-or-silence mapping of text tokens. The authors report over 50% fewer duplicate n-grams than standard GRPO.
• Multi-reward DPO for blockwise duplex. Semantic, audio-quality and emotion-consistency rewards aggregated in one DPO objective, studied on blockwise duplex models (arXiv 2601.19063).

Algorithms

GRPO dominates for codec LMs: sample a group of $G$ rollouts per prompt, normalise rewards within the group, and keep a KL penalty to the SFT reference. Standard notation for the advantage of rollout $i$:

$$\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

With several rewards, a common choice is to normalise each reward within the group before summing. Inference GROW (arXiv 2608.03215) standardises its rewards within the group; I have not checked other papers.

Reward hacking: cases and guards

Inference The guards in the right column are my pairing of cases with mitigations, except where a source is named.

Infrastructure and cost

Every rollout must be decoded to a waveform, then passed through an ASR and an SV model. Inference A SpeechJudge-style reward model dominates the cost. Cheaper options are token-level rewards, cached reference embeddings and a small ASR for the reward.

• FunAudioLLM/CosyVoice: GRPO training support and an RL model, per the README.
• zai-org/GLM-TTS: multi-reward GRPO TTS README.
• AmphionTeam/SpeechJudge: the generative reward model.
• Kyutai RL checkpoints: kyutai/moshika-rl-seamless, kyutai/personaplex-rl-seamless.

Bridge from GRPO on streaming ASR

Practical recipes

Monitor: per-reward mean and variance, fraction of zero-advantage groups, held-out metrics, F0 spread, silence ratio, speaking rate, duration distribution, KL, clip fraction and end-of-sequence behaviour. Stop when held-out metrics degrade even if the train reward rises, and spot-check audio.

Evaluating conversational systems

Evaluation moved from demos to automated overlap benchmarks (Full-Duplex-Bench) and annotated dual-channel benchmarks (TurnBench, arXiv 2608.25218, Aug 2026). Latency alone is a weak number: it only means something at a stated false-interruption rate. The open problem, per the TurnBench authors, is that no current system matches human timing without excessive false positives.

Latency definitions and operating points

LiveKit's eot-bench (LiveKit blog) reports two views of the same detector: the false-cutoff rate at a fixed latency budget (300 or 600 ms), and the mean latency at a fixed false-cutoff target (5% or 10%). Reported For LiveKit's own v1, the blog gives 9.9% false cutoffs at 300 ms and 4.5% at 600 ms, and 543 ms mean latency at 5% and 295 ms at 10%. It is a vendor benchmark, but the operating-point framing is reusable for any turn-taking system.

Vendor latency claims are not comparable. Deepgram's Flux docs state "~260ms end-of-turn detection" inside the STT model, without a stated benchmark. That cannot be compared with the eot-bench operating points above.

Benchmarks

Reported numbers

• TurnBench. Confirmed The VAP baseline scores end-of-turn recall 0.845, false-positive rate (FPR) 0.055, median latency 368 ms; interruption recall 0.945, FPR 0.107, median 994 ms. Humans on smooth transfers start a median 151 ms before the current turn ends.
• DuplexSLA-Bench (arXiv 2605.20755). Reported Authors' backchannel accuracy, context-prefill setting: gpt-realtime-1.5 0.33% (semantic VAD), gemini-3.1-flash-live 40.00%, DuplexSLA 98.33%.
• Instruct-FD (arXiv 2607.20460). Reported The authors report that the best of six full-duplex systems achieves only 64.4% adherence to turn-taking instructions.
• PersonaPlex (arXiv 2602.06053). Reported The authors report 0.170 s smooth turn-taking and 0.240 s user-interruption latency for the released 7B checkpoint, against 0.265 s and 0.257 s for Moshi.

Ten proposed latency-aware and interaction-based metrics

Inference This list is my proposal, built on the benchmarks above. Every timing metric uses gold event labels and echo-cancelled agent output, never the system's own VAD.

The set separates timing from content, in line with DuplexPO, and ties latency to an operating point.

Steering voice, tone and emotion

Five control channels, usually combined: a reference clip, a natural-language instruction, inline tags, duration targets, and style guidance at sampling time.

Control channels

Disentanglement

IndexTTS2 separates emotion from timbre: timbre comes from one prompt and emotional tone from a style prompt, so the two can come from different speakers (arXiv 2506.21619). Its text-based emotion control comes from fine-tuning Qwen3. The authors report better word error rate, speaker similarity and emotional fidelity than other zero-shot TTS models. Step-Audio-EditX takes a different route: large-margin synthetic data instead of representation-level disentanglement (arXiv 2511.03601).

In CosyVoice's own tests, the instruct model raised the share of generated speech that an emotion2vec classifier labelled sad, angry, surprised and disgusted, compared with the base model (arXiv 2407.05407, Table 10).

Duration control and lip-sync

The IndexTTS2 authors motivate exact durations by video dubbing, where speech must match a video block length (arXiv 2506.21619). Inference This is the hook for avatar sync.

Evaluation

emotion2vec (arXiv 2312.15185) is a self-supervised speech-emotion representation, usable as the basis for emotion similarity. Synthesia reports measuring Express-Voice with WavLM for speaker similarity and emotion2vec for emotional similarity.

Synthesia context

Synthesia's Express-2 (4 Sep 2025) pairs Express-Voice with Express-Video. The post reports a blind test with 100 native English-speaking evaluators across 17 accents, in which Express-Voice rated highest for matching the original speaker's identity, rhythm and accent (Reported company claim). The post gives no scores.

Post-training and evaluation for speech generation

Post-training uses ASR and speaker-verification rewards. The recurring failure is reward hacking.

Methods

Reward hacking

Metrics

Link to my work

The GRPO experience from online RL at Amazon transfers directly. The new part is making rollouts differentiable or likelihood-tractable for flow models.

Audio-visual diffusion

Three threads have converged. Large DiT video backbones now take audio as a condition or generate it jointly. Avatar models bolt audio conditioning and long-video stitching onto those backbones. Causal few-step distillation turns the result into a streamer.

Real-time numbers in this section are author-reported, often multi-GPU, and not comparable across papers.

Video diffusion backbones

The shared recipe is a 3D causal VAE, a DiT denoiser, and a flow-matching or rectified-flow objective Inference. Models differ in attention sparsity, text encoders, and how causal the VAE is.

Audio-driven human animation

These models take a reference image plus audio and render a talking person. Most reuse a video foundation model (Wan, Hunyuan) and add audio cross-attention Inference.

• OmniHuman-1 (arXiv 2502.01061): a DiT that scales up data by mixing motion-related conditions into training. OmniHuman-1.5 (arXiv 2508.19209): an MLLM writes a structured textual representation for semantic and emotional guidance; a "Pseudo Last Frame" design mitigates inter-modality conflicts. The authors argue earlier models sync to low-level cues such as audio rhythm and lack understanding of emotion and intent.
• HunyuanVideo-Avatar (arXiv 2505.20156): an MM-DiT for multi-character, emotion-controllable dialogue video. A face-aware audio adapter isolates each character with a latent-level face mask, so audio is injected per character through cross-attention.
• FantasyTalking (arXiv 2504.04842): audio-to-video in a pretrained video DiT, then lip-focused refinement. FantasyTalking2 (arXiv 2508.11255): preference optimisation (TLPO) with a reward model, Talking-Critic, and a 410K-pair dataset.
• MultiTalk (arXiv 2505.22647): multi-person conversation with per-stream audio; Label RoPE (L-RoPE) resolves which audio stream binds to which person. InfiniteTalk (arXiv 2508.14033): sparse-frame dubbing for infinite-length sequences, editing full-body motion rather than only the mouth region.
• EchoMimicV3 (arXiv 2507.03905): a 1.3B-parameter framework that unifies multi-task and multi-modal human animation.
• Synthesia Express-2 (4 Sep 2025): Express-Voice, Express-Animate (co-speech motion from audio), Express-Eval and Express-Render (1080p, 30 fps, arbitrary length). Express-Render is distilled to two diffusion steps. The post makes no real-time claim Reported.

Joint audio and video generation

Joint models replace post-hoc lip sync by generating both streams together. Audio quality is still the open question.

Ovi (Character.AI, arXiv 2510.01284) is the clearest open recipe.

• Twin DiT towers: video initialised from Wan2.2 5B; the audio tower has the same shape (30 blocks, dim 3072) and trains from scratch.
• Bidirectional audio-video cross-attention in every block.
• RoPE on both streams, with the audio frequencies scaled to match the coarser video resolution. Video latents span 31 frames while audio latents form 157 tokens.
• The authors note that dense blockwise fusion costs time per sampling step, and name distillation such as DMD2 as a way to reduce steps.

UniVerse-1 (arXiv 2509.06155) takes the other route. It stitches pretrained experts, Wan2.1 (1.3B) for video and ACE-Step (3.5B) for music, with lightweight cross-modal MLP connectors in corresponding blocks. It was fine-tuned on about 7,600 hours of audio-video data.

Coupling therefore comes in two patterns: same-shape twin towers with per-block cross-attention (Ovi), or pretrained experts joined by connectors (UniVerse-1). Ovi handles the token-rate mismatch with RoPE rescaling.

Streaming and real-time

The recipe is to distil a bidirectional teacher into a causal few-step student with a KV cache. The failure mode is error accumulation over long rollouts (CausVid, Self Forcing and AvatarForcing all discuss it).

Error accumulation has several named causes and fixes. The train-test gap is closed by training on own rollouts (Self Forcing). LiveTalk reports flicker and black frames when Self Forcing meets multimodal conditioning, and repairs them with a better distillation recipe. Identity drift is handled with identity sinks (LiveTalk) and a reference sink (StreamAvatar). Mechanisms are rolling or sink KV caches, on-policy rollouts and distribution matching.

Decoupled Self-Forcing (arXiv 2609.10317) frames two routes: end-to-end audio-conditioned video diffusion (high quality, but only at large scale) against two-stage methods that generate an intermediate motion representation (cheaper, lower fidelity). Its own approach pairs a small causal AR motion generator with a diffusion renderer and reports 15.4 FPS at 1.3 s latency. SoulX-FlashTalk (arXiv 2512.23379) argues that strictly unidirectional attention often compromises visual fidelity.

Metrics and data

• Lip sync: Ovi's data filter uses SyncNet to filter out clips lacking audio-video synchronization (arXiv 2510.01284); UniVerse-1 reports LSE-C (arXiv 2509.06155).
• Identity: UniVerse-1 scores identity as mean DINOv3 feature similarity between the reference image and each generated frame (arXiv 2509.06155).
• Datasets named in the papers above: HDTF, CelebV-HQ and EMTD (InfiniteTalk), and HDTF, AVSpeech and CelebV-HQ (LiveTalk).
• Human preference studies (Ovi ran a blind pairwise study with 50 participants) and reward-model judges (Talking-Critic, arXiv 2508.11255) appear in the papers above.
• Ovi's curation also filters clips on a minimum mean volume and adds captions (arXiv 2510.01284).

Autoregressive and unified models

The field has moved from a discrete-versus-continuous fight to hybrids. Next-token AR dominates unified stacks, but the per-token head is increasingly diffusion or flow over continuous latents. Video is moving to chunked, causal AR with diffusion for streaming.

Discrete-token AR

• VAR (arXiv 2404.02905): next-scale prediction. The authors report ImageNet 256 FID improving from 18.65 to 1.73 against an AR baseline, with around 20x faster inference.
• Infinity (arXiv 2412.04431): bitwise token prediction with an infinite-vocabulary tokenizer and bitwise self-correction. The authors report GenEval 0.62 to 0.73 over SD3-Medium, and a 1024 px image in 0.8 s, 2.6x faster than SD3-Medium.
• Emu3 (arXiv 2409.18869): one transformer trained from scratch with next-token prediction over discrete image, text and video tokens. The authors claim it surpasses SDXL and LLaVA-1.6.
• Emu3.5 (arXiv 2510.26583): world-model framing on interleaved vision-language data, over 10T tokens, RL post-training. Discrete Diffusion Adaptation (DiDA) turns token-by-token decoding into bidirectional parallel prediction, accelerating per-image inference by about 20x (authors).

Continuous-token AR with diffusion heads

• MAR (arXiv 2406.11838, NeurIPS 2024): no vector quantisation; each token's distribution is modelled with a diffusion procedure (Diffusion Loss), including masked autoregressive variants.
• Fluid (arXiv 2410.13863): continuous tokens give clearly better visual quality than discrete; random order beats raster on GenEval. At 10.5B: zero-shot FID 6.16 on MS-COCO 30K, GenEval 0.69.
• NOVA (arXiv 2412.14169, ICLR 2025): non-quantised AR video, frame by frame in time and set by set within a frame; 0.6B parameters.

Chunked AR video diffusion

• MAGI-1 (Sand AI, arXiv 2505.13211): autoregressively predicts fixed-length video chunks, trained to denoise per-chunk noise that increases monotonically over time, which supports streaming generation. The largest variant has 24B parameters and supports context lengths up to 4 million tokens (authors).
• Self Forcing (arXiv 2506.08009, NeurIPS 2025): trains on its own rollouts with a rolling KV cache. The authors report 17 FPS with sub-second latency on one H100. Inference It is the most relevant entry for interactive avatars.

Hybrid unified models

One related observation from the Kelix report: most VLMs pair discrete text tokens with continuous ViT features, which biases them toward understanding (arXiv 2602.09843).

Discrete diffusion language models

• LLaDA (arXiv 2502.09992): 8B masked-diffusion LM trained from scratch; the authors report it is competitive with LLaMA3 8B in in-context learning.
• MMaDA (arXiv 2505.15809): a multimodal diffusion language model with UniGRPO, a unified policy-gradient RL algorithm for post-training across reasoning and generation.

The convergence debate

• Continuous tokens with a diffusion or flow head give better per-token quality (Fluid, MAR). Discrete tokens give a simpler LM interface and a single cross-entropy (Emu3, LLaDA) Inference.
• Random-order or bidirectional-within-block generation is winning for images. Raster order is kept where causal streaming matters Inference.
• Chunked, causal AR suits streaming video (MAGI-1, Self Forcing). Full-clip diffusion remains the offline quality leader Inference.

Tokenizers beyond speech

Across images, video and audio, the tokenizer matters more than the generator. The consensus design is a causal, heavily compressed latent, regularised toward semantic features Inference. Discrete codecs stay the default for audio LMs; continuous VAE latents stay the default for diffusion and flow.

Image and video tokenizers

• MAGVIT-v2 (arXiv 2310.05737, ICLR 2024) uses lookup-free quantisation (LFQ). With this tokenizer, the authors show LLMs outperforming diffusion models on ImageNet and Kinetics, and title the paper "Tokenizer is Key to Visual Generation" Reported.
• FSQ (arXiv 2309.15505) quantises each latent dimension to a small set of fixed values. The codebook is the implicit product of those sets, so the authors report no codebook collapse and no need for commitment losses or codebook reseeding Confirmed.
• TiTok (arXiv 2406.07550) is a 1D tokenizer. It reduces a 256x256x3 image to 32 discrete tokens, with gFID 1.97 on ImageNet 256 using the same generator framework Reported.
• The NVIDIA Cosmos tokenizer outputs either continuous latents or discrete tokens for images and video. Its README states total compression of up to 2048x (8x16x16) and running up to 12x faster than the best available tokenizers Reported.
• Causal 3D VAEs compress time and space without peeking at future frames. Wan's VAE replaces GroupNorm with RMSNorm to preserve temporal causality, which enables a feature cache (arXiv 2503.20314) Confirmed.

FSQ in standard notation (not taken from the sources): bound each latent dimension, then round.

Here $L_i$ is the number of levels in dimension $i$, and $|\mathcal{C}|$ is the implicit codebook size.

Semantic-aligned VAEs

A recent trend is to align the latent with, or build it directly on, vision foundation model features, then decode.

General audio and music codecs

Unified and joint audio-visual tokenizers

• UniCodec (arXiv 2502.20067): one codec with a single codebook across speech, music and sound, using a partitioned domain-adaptive codebook and domain Mixture-of-Experts.
• MelTok (arXiv 2510.01903): a 2D mel-spectrogram tokenizer that compresses 44.1 kHz audio into a single codebook, with a token-based vocoder.
• EntangleCodec (arXiv 2606.02739): a unified discrete audio tokenizer with caption-aligned semantic-acoustic representations and a flow-matching diffusion decoder.
• AVTok (arXiv 2606.30811): a 1D joint audio-video tokenizer with a shared encoder-decoder, modality-specific learnable queries and a unified codebook Reported.
• VideoPoet (arXiv 2312.14125) is the precedent: it concatenated MAGVIT-v2 and SoundStream vocabularies into one unified vocabulary rather than learning a joint tokenizer.

Trade-offs

The table is directional, not a benchmark Inference.

Bridge to speech codecs

Speech codecs (Mimi, X-Codec2, BiCodec, CosyVoice 2 FSQ tokens) are covered in chapter 03. The same logic applies there: low rate and semantic distillation help the LM, continuous latents win back fidelity. Inference For a Voice team the new point is that audio-visual tokenizers may be converging on the same recipe as speech ones.

Multimodal LLM architecture

Open multimodal LLMs (Qwen3-VL, InternVL3.5, Llama 4) share one template: a vision encoder, a token-reducing connector, and a decoder LLM with multimodal positional encoding. The differences are in positional encoding, visual token count and MoE. Each claim below is cited to the model's technical report, its arXiv paper or Meta's own post.

Encoder, connector, LLM

• Qwen3-VL uses DeepStack to integrate multi-level ViT features, plus an enhanced interleaved MRoPE and explicit text timestamps for video. It has dense 2B/4B/8B/32B and MoE 30B-A3B and 235B-A22B variants, with native 256K context (arXiv 2511.21631) Confirmed.
• Its vision encoder is SigLIP-2, a two-layer MLP merger compresses visual features into tokens, and DeepStack routes visual tokens from several ViT layers into corresponding LLM layers through lightweight residual connections (arXiv 2511.21631) Confirmed.
• InternVL3.5 compresses 1024 visual tokens per image patch to 256 with a pixel shuffle module. Its Visual Resolution Router (ViR) dynamically adjusts the resolution of visual tokens (arXiv 2508.18265) Reported.
• Llama 4 uses a MetaCLIP-based vision encoder trained separately in conjunction with a frozen Llama, with MoE (Scout: 17B active, 16 experts, 109B total; Maverick: 17B active, 128 experts, 400B total; Meta's Llama 4 post) Reported.

Positional encoding

Video is handled as time-indexed positions. That one choice enables audio-video sync and streaming Inference.

• Qwen3-Omni's TM-RoPE splits rotary dimensions into temporal, height and width (24/20/20 angles). Audio gets one temporal ID per 80 ms. Video frame IDs are adjusted to real timestamps (arXiv 2509.17765 section 2.3) Confirmed.

Early vs late fusion

Meta describes Llama 4 as using early fusion to integrate text and vision tokens into a unified model backbone (Meta's Llama 4 post) Reported.

Omni models and time alignment

Qwen3-Omni is a Thinker-Talker MoE (arXiv 2509.17765). The AuT audio encoder runs at a 12.5 Hz token rate. The Talker generates one codec frame per step, an MTP module produces the residual codebooks, and a lightweight causal ConvNet (Code2Wav) renders the waveform Confirmed.

Qwen3-Omni's theoretical first-packet latency in cold-start settings is 234 ms for audio and 547 ms for video. At 4 concurrency, Table 2 gives 728 ms and 1517 ms Confirmed. These are theoretical figures, not end-to-end measurements.

Training and RL

• InternVL3.5 uses Cascade RL: offline RL for stable convergence, then online RL for refined alignment (arXiv 2508.18265) Reported.
• A study of R1-like training for vision-language models reports that SFT before RL can undermine RL by inducing imitated "pseudo reasoning paths". Its GRPO variant with a mixed perception and cognition reward fostered more adaptive reasoning (arXiv 2504.11468) Reported.

Streaming video understanding

StreamingVLM (arXiv 2510.09608) keeps a fixed-size KV cache of attention sinks, recent vision tokens and recent text tokens. It uses contiguous RoPE. It runs at up to 8 FPS on one H100 on a Qwen2.5-VL-7B base Confirmed. Authors report a 66.18% win rate against GPT-4o mini on Inf-Streams-Eval and +4.30 on LongVideoBench Reported. Related: ReKV (arXiv 2503.00540) stores video KV caches in RAM and disk and retrieves only the query-relevant ones.

Evaluation caveat

What an avatar that perceives the user needs

Speech-LLM design, the audio half of this stack, is in chapter 02 and chapter 04.

Speech generation problems

Thirteen failure modes recur in TTS and voice cloning. Most come from three places: weak text-to-audio alignment, conditioning on a flawed reference, and stochastic sampling. The table is the quick map; the subsections below cover the ones worth explaining aloud.

Grounding labels follow the rest of the page. Synthesia facts cite its Express-2 post and its docs; research claims cite arXiv. Mechanisms marked Inference are my reading, not a source. The Metric and Mitigation columns of the table are suggestions unless a subsection cites them.

A1 Hallucination and alignment

LLM-based TTS models can repeat words, drop words or produce misaligned speech Reported. The authors of arXiv 2406.17957 find that certain cross-attention heads implicitly learn the text-to-speech alignment, and that nothing in the loss forces that alignment to stay monotonic. Regotron adds a monotonic alignment loss to Tacotron 2 for the same class of problem (arXiv 2204.13437).

GOAT reports a strong positive correlation between hallucination and model uncertainty, and over 50% lower character error rate on challenging test cases (arXiv 2508.15442) Reported.

Flow-matching TTS is non-autoregressive, so it has no free-running attention to stall (see diffusion and flow). The cost is that the total duration must be chosen before generation, and that choice can be wrong Inference.

A3 Mispronunciation

Names, acronyms, numbers and heteronyms fail when grapheme-to-phoneme rules or text normalisation guess wrong Inference. Express-Voice operates on graphemes directly, so any fix has to live in the model or in the script Confirmed (Express-2 post).

Synthesia's documented workaround is a per-word pronunciation entry Confirmed. The script docs give a respelling example, "sin-THEE-zhuh" for Synthesia, and offer two previews: an IPA version and a respelling that replaces the text before it reaches the voice model. Constraints from the docs and the help article:

• A saved pronunciation does not automatically apply to instances of the word added to the script later.
• "Apply to all" covers every instance in the current video.
• "Add to glossary" applies across the workspace and is available on Enterprise plans only.
• Pronunciation reliability varies by voice and language; regenerating the scene can help.

The pattern is a manual patch around a grapheme model. A production fix would add a per-customer dictionary applied before synthesis and a fixed name list for regression tests Inference.

A2 Long-form drift

Over minutes, autoregressive generation can lose speaker identity or flatten prosody because errors compound and nothing re-anchors the voice Inference. Measure speaker similarity in sliding windows against the first window, and plot it against time. Mitigate by synthesising in chunks and re-conditioning each chunk on the original reference.

A6 and A8 Reference leakage

Express-Voice clones in context: the reference audio conditions the model with no fine-tuning Confirmed (Express-2 post). The same mechanism that copies timbre can copy emotion, background noise and microphone channel Inference.

• Detect leakage with emotion2vec similarity between prompt and output, and a noise-floor check.
• Gate prompts at upload: reject clipped, noisy or reverberant audio, or denoise first.
• Prefer neutral, clean references when the target style is set by the script.

A7 Metric pitfalls and A11 codecs

Synthesia names WavLM for speaker similarity and emotion2vec for emotional similarity in its voice evaluation, and publishes no scores Confirmed (Express-2 post). A similarity score depends on which speaker encoder computes it, so scores from different encoders do not compare Inference.

Express-Voice uses Descript's residual vector-quantised tokenizer Confirmed. A metallic or buzzy timbre that survives every sampling setting points at the tokenizer, not the LM Inference. See codecs and tokenizers for the rate trade-offs.

A9 Sampling instability and reranking

Higher temperature plausibly raises the hallucination rate of A1, so a few samples fail even when the average looks good Inference. The cheap fix is best-of-N: sample several candidates, score each, return the best. Score with ASR error plus a speaker-similarity term.

Synthesia applies the same idea to video: Express-Eval scores audio-motion alignment and picks the best motion candidate Confirmed (see audio-visual problems). Pushing the reranking signal into training is online RL, which invites reward hacking. One report finds that GRPO on transcription signals (CER and NLL) lowers error rates but collapses prosody into monotone speech (arXiv 2509.18531) Reported. See online RL rewards for composite rewards and KL control.

A4, A5, A10, A12, A13 in brief

• Accents. Express-Voice training data was curated for accent coverage, including YODAS and LibriLight. Synthesia reports a blind test with 100 native English-speaking evaluators across 17 accents and top ratings, but names no comparison systems and gives no scores Reported (Express-2 post). Report per accent and per language pair Inference.
• Prosody. Express-2 offers voice styles for the same voice (for example Excited and Neutral) Confirmed. The script docs add a pause tool with durations from 0.1 s to 99 s (default 1 s) (script docs). That is coarse control, and a flat read needs a script edit Inference.
• Chunk seams. Streaming trades first-packet time against boundary quality Inference. See streaming and low latency.
• Languages. Synthesia's own pages disagree on language counts. The Synthesia 3.0 post says dubbing covers 30+ languages, while the title of its AI dubbing page says 130+. Quote the number only with its page.
• Misuse. Synthesia states that it strictly prohibits cloning voices without explicit consent Confirmed (voice cloning page).

Speech-to-speech and duplex problems

End-to-end and full-duplex models add failures that cascades do not have. The model must listen while it speaks, decide when to talk, and keep its text intelligence while operating on audio tokens. Synthesia's Video Agents "talk, listen, and act in real time" inside a video Confirmed (Synthesia 3.0 post), so these failures reach its interactive product, not only research systems.

Intelligence drop and forgetting

Adding speech capability can degrade an LLM's knowledge and reasoning even when inputs stay textual, and performance drops further on spoken queries (arXiv 2509.14930) Reported. The SALAD authors name the resulting shortfall the "text-speech understanding gap" (arXiv 2510.13632). Two recipes exist: freeze the text LLM (Freeze-Omni, arXiv 2411.00774), or keep a high share of text in the training mix (Moshi trains half of the time on text-only batches, arXiv 2410.00037; Kimi-Audio pretrains on text-only, audio-only and interleaved data, arXiv 2504.18425).

Interleaved synthetic speech-text data lifted spoken question answering from 13% (Moshi) to 31% in one report (arXiv 2411.17607) Reported. Gate every stage on a strict text QA score so forgetting shows up early Inference. See streaming conversation.

Multi-turn degradation

Full-Duplex-Bench v2 tests daily, correction, entity-tracking and safety task families. Its authors report that the tested systems often get confused when people talk at the same time, struggle with corrections and sometimes lose track of who or what is being discussed (arXiv 2510.07838) Reported. FLEXI reports significant gaps between open-source and commercial models in emergency awareness, turn terminating and interaction latency (arXiv 2509.22243) Reported.

Backchannel misread as barge-in

A 2026 survey calls this the single most-cited failure mode in user feedback: the system mistakes the user's acknowledgement ("uh-huh") for a barge-in and stops (arXiv 2606.19453) Reported. Talking Turns finds that audio foundation models can interrupt too aggressively and rarely backchannel themselves (arXiv 2503.01174). The open problem is the trade-off between responding early and yielding falsely Inference.

DuplexSLA reports backchannel accuracy of 98.33%, against 40.00% for gemini-3.1-flash-live and 0.33% for gpt-realtime-1.5 with semantic VAD (arXiv 2605.20755, Table 6) Reported. These numbers are self-reported by the DuplexSLA authors on their own scenarios. Quote them as a claim, not a ranking.

Self-echo, latency spikes and quantisation

• Self-echo. A model that ingests user audio while speaking can hear its own voice. Mitigate with acoustic echo cancellation and by conditioning on the model's own output stream Inference.
• Chunk quantisation. SyncLLM alternates 160 ms chunks of user and assistant tokens (arXiv 2409.15594), and the survey calls it an example of "apparent (not substantive)" full duplex, since transitions occur only at chunk boundaries (arXiv 2606.19453) Reported. Turn-taking feel plausibly suffers when decisions are quantised at 80 to 240 ms, trained on heuristic labels, or starved of backchannels in the data Inference.
• Latency spikes. Reported latencies are measured under different conditions and do not compare Inference. Track p95 and p99, because a conversation is judged by its worst pauses. See streaming and low latency.

Interruption recovery and truncation

On barge-in the agent must stop, and its context must record only the words actually spoken. Otherwise it believes it said a sentence the user never heard. Measure stop latency from the gold onset to the last audible agent sample on echo-cancelled output, plus the fraction of agent speech after onset Inference.

Long-session context and synthetic data

Token-level duplex models grow the KV cache continuously. A back-of-envelope for Moshi's temporal transformer uses its published shape of 32 layers and 4096 dimensions (arXiv 2410.00037) and assumes full multi-head attention in bf16: $2 \times 32 \times 4096 \times 2$ bytes is about 512 KiB per frame, or roughly 23 GB per hour at 12.5 frames per second Inference. Grouped-query attention shrinks it.

Duplex training data is often synthetic: an LLM writes dialogue and TTS renders both sides. SyncLLM trains on 212k hours of synthetic dialogue and 2k hours of real speech (arXiv 2409.15594), and PersonaPlex on synthetic conversations from open LLMs and TTS (arXiv 2602.06053) Reported. Risks are TTS voice and prosody leakage, over-regular inserted backchannels, and circularity, since inserting backchannels needs dialogue that already contains them Inference. DuplexPO applies RL over selected windows of human conversations to learn when to speak separately from what to say (arXiv 2607.07148) Reported. See online RL rewards for per-axis timing rewards and the hacking risks.

Audio-visual problems, grounded in Synthesia

Synthesia's Express-2 stack (September 2025) has four parts: Express-Voice for speech, Express-Animate for motion, Express-Eval for motion scoring, and Express-Render for frames Confirmed (Express-2 post, 4 September 2025). Each stage can introduce its own failures, and later stages cannot always repair them. The figure shows where each problem (B1 to B11 and A-codes from the previous section) enters; the placement is my mapping Inference.

B1 Lip-sync, and why Sync-C misleads

Hard cases plausibly include bilabial closures (p, b, m), fast speech and non-English audio Inference. The standard metrics are Sync-C and Sync-D, also written LSE-C and LSE-D, which are based on SyncNet (arXiv 2403.06421). Evidence that they mislead:

• THEval's authors say existing metrics show limited correlation with human evaluation and unreasonable values, such as ground truth being widely outperformed (arXiv 2511.04520).
• In a psychophysical comparison of metrics on talking-head videos, only LSE-O showed moderate effectiveness (arXiv 2403.06421).

Both are Reported results from papers. One proposed replacement uses an audio-visual speech representation expert (AV-HuBERT) for a lip-sync loss and three new evaluation metrics (arXiv 2405.04327). Sync-C values from different papers use different protocols, so I treat cross-paper tables as unreliable Inference. See metrics and data.

B2, B3 Identity drift, flicker and colour drift

Express-Render is described as keeping identity consistent over long sequences, with no metric attached Confirmed (Express-2 post). A related evaluation framework studies lip leakage, where the identity reference image influences the lips instead of the driving audio (arXiv 2511.08613) Reported. Open avatar work offers fixes for long sequences: sparse-frame conditioning to preserve identity (InfiniteTalk, arXiv 2508.14033) and a multi-step retrospective self-correction mechanism against accumulated error (SoulX-FlashTalk, arXiv 2512.23379) Reported. Flicker around teeth and mouth interior is a plausible weak spot Inference. The mechanisms are explained in streaming and real-time.

B4, B5, B6 Gesture and expressiveness

Express-2 avatars combine facial expressions and lip sync with natural hand and body gestures, and the post publishes no failure rates Confirmed. Express-Animate generates gestures from audio alone, which decouples motion from appearance Confirmed. Audio-only conditioning plausibly yields rhythm without intent Inference. OmniHuman-1.5 makes the same argument: earlier avatar models synchronise with low-level cues such as audio rhythm and lack semantic understanding of emotion and intent, so it adds MLLM-written semantic guidance (arXiv 2508.19209) Reported.

Express-2 also exposes performance intensity and temporal diversity controls Confirmed. To measure voice-face mismatch, compare emotion2vec on the audio with a facial-expression classifier on the video Inference.

B7 Dubbing

Synthesia's dubbing translates uploaded video into 30+ languages with frame-accurate lip-sync Confirmed (Synthesia 3.0 post). The post gives no lip-sync or voice-retention metrics.

The hard constraint is duration: translated speech is often longer than the source line, so the system must shorten the translation, speed the speech, or retime the video Inference. Measure the fraction of segments outside a timing budget, then lip-sync quality after retiming.

B8, B9 Streaming, resolution and compression

The Express-2 post gives no latency numbers. It says generation times should stay similar after the two-step distillation, and describes Express-Render-Turbo as coming later Confirmed. The quality versus latency trade-off is unpublished Inference.

Causal streaming students show their own failures. LiveTalk reports that Self Forcing, the leading on-policy distillation approach, shows flickering, black frames and quality degradation under multimodal conditioning, and proposes an improved distillation recipe (arXiv 2512.23576) Reported. Error accumulation plausibly grows with rollout length, and audio and video clocks can drift apart Inference. Output is 1080p at 30 fps with no stated duration limit Confirmed. Because encoder settings can change metric values, pin the encode settings before scoring Inference.

B11 Safety

Synthesia requires a live consent video that must show the same person as the photo, and does not accept uploaded consent videos Confirmed (personal avatars docs). It states that it strictly prohibits cloning voices without explicit consent (voice cloning page) and moderates content at the point of creation Confirmed. Its AI governance page lists a Content Integrity Policy, EU AI Act Article 50 labelling and Content Authenticity Initiative engagement. The page does not mention C2PA or watermarking.

Express-2 component to problem map

Priorities, metrics and questions

Not every failure deserves equal effort. This ranking weighs how visible the failure is to a customer, how often it occurs in a voice and avatar product, and how cheaply it can be measured. The ranking, mitigations and metrics are my own judgement Inference, built on the sourced problems in the previous sections.

Why this order

• Items 1 and 2 are audible on first listen and cheap to test automatically, so they belong in every regression run.
• Items 3 to 7 are the avatar-specific problems; they need human rating because the automatic metrics are weak.
• Items 8 to 10 depend on product shape (dubbing, streaming, Video Agents) and rank higher if that product is the focus.
• Any metric that is optimised in training needs a held-out judge, or reward hacking will hide regressions (see online RL rewards).

Questions to ask Synthesia about failure modes

1. Which failure modes do you track in production, and which of them have an automatic metric?
2. How do you evaluate lip-sync, given the known weaknesses of Sync-C, and does it differ for non-English dubbing?
3. What does Express-Eval catch in practice: gesture plausibility, lip-sync, or both, and how often does it change the pick?
4. For Express-Voice, how do you protect against emotion or noise leakage from customer reference audio?
5. For Video Agents, what is the p95 first-frame latency, and how do you handle barge-in and backchannels?

ML system design: a unified speech LLM with a downstream reasoning LLM (the target)

Why a unified speech LLM, and why a separate reasoning LLM

Inference The honest costs of the target: an extra hop of LLM time to first token; paralinguistic information must be flattened into the call, so what the tags omit the LLM never sees; and two models to serve, version and evaluate together.

Precedents for each piece. Kyutai DSM applies one architecture to both jobs: ASR delays the text behind the audio (2.5 s default), and TTS delays the audio behind the text (1.28 s), with the delay set per stream Confirmed (arXiv 2509.08753). The DSM authors report a 150 ms first-chunk latency for TTS at batch size 1 Reported. Step-Audio 2 calls external tools such as web search against hallucination and audio search to switch timbres Confirmed (arXiv 2507.16632). Qwen3-Omni uses a Thinker-Talker design, the in-model analogue of this split (arXiv 2509.17765). Freeze-Omni keeps the LLM frozen and adds a state classifier (keep listening, interrupted, turn ended) (arXiv 2411.00774). MinMo adds an 18M-parameter Full Duplex Predictor beside the LLM (arXiv 2501.06282). Both show a small decision module beside a text LLM works. Moshi's Inner Monologue shows text and audio can share one clock (arXiv 2410.00037). Confirmed

Requirements

Architecture

• Listen head. A causal streaming encoder at 12.5 Hz feeds a text-LLM-initialised backbone. The head emits streaming text (partials and stable finals), paralinguistic tags (emotion, rate, language, confidence) and an action token per frame. The user stream is ingested continuously, including while the agent speaks.
• Tool call. On an end-of-turn decision the head emits a structured call: the stable transcript, the tags, the turn state (end-of-turn confidence, whether it interrupted a reply) and a pointer to what was actually played last. Start on the end-of-turn candidate and cancel on a resumed turn, as in the cascade baseline.
• Downstream LLM. Any text LLM with tools and retrieval. It streams text back with style cues (emotion, pace) the speak head can use. It never touches audio, so it can be swapped or upgraded without retraining the speech model.
• Speak head. It turns text chunks plus style and the voice prompt into codec tokens at 12.5 Hz (semantic then acoustic codebooks through a depth transformer or multi-token head), decoded by a causal codec decoder. A small first chunk cuts time to first audio; later chunks carry more lookahead for prosody.
• Latency cover. While the call runs, the speak head says a short acknowledgement or filler, and the listen head can backchannel during long user turns. Neither waits for the LLM.
• One clock. Every output chunk carries an absolute timestamp (Qwen3-Omni assigns one temporal ID per 80 ms, arXiv 2509.17765). Confirmed The renderer and client play audio on that clock and shed video frames first.

Conversational behaviours: who decides what

Inference The speech LLM owns timing and the floor; the text LLM owns content. The action channel is one token per frame or chunk: listen, end-of-turn, backchannel, speak, yield. DuplexSLA is the precedent for such a channel (a textual action channel on a shared 160 ms chunk clock; 98.33% backchannel accuracy, self-reported; arXiv 2605.20755), and Freeze-Omni and MinMo show the same decision made by a small module beside a text LLM. Reported

Definitions and operating points are in conversational evaluation; the behaviour list follows backchannels, barge-in and self-voice. The human anchor is that listeners begin speaking a median 151 ms before the other turn ends (TurnBench, arXiv 2608.25218), and a decision at a chunk boundary waits $c/2$ on average, so decide per 80 ms frame or short chunk. Confirmed (anchor) Inference (the design reading)

Key design decisions

Data

• Listen head. Large unlabelled audio with pseudo-transcripts (Moshi uses 7M h, arXiv 2410.00037; Kimi-Audio over 13M h, arXiv 2504.18425), plus ASR data to align the encoder. Confirmed Paralinguistic tags come from pseudo-labellers (Synthesia names emotion2vec in its Express-Voice evaluation), checked on a human-labelled subset. Inference
• Speak head. TTS-grade, speaker-diverse, consented speech with style labels; interleaved synthetic speech-text (1T tokens, 600B synthetic interleaved; spoken QA rose from 13% to 31% in arXiv 2411.17607). Confirmed
• Action channel. Dual-channel conversation with real overlap: Fisher, about 2,000 h of phone conversations (arXiv 2410.00037), MinMo's 4,000 h of duplex data (arXiv 2501.06282), and Seamless Interaction conversations (used by Kyutai for RL, arXiv 2606.11167). Confirmed Synthetic dialogue has few real overlaps and regular backchannels, so mix real and synthetic.
• Tool-call traces. An LLM writes dialogues with interruptions; a harness renders both sides with many TTS voices and logs the call, the reply and the truncation. Inference The baseline cascade is the generator and the first labeller.
• Consent. Every voice clip carries a consent record and licence scope; withdrawn voices are removable from retrains. Inference

Training stages and rewards

1. Align the encoder and adaptor on ASR with the LLM backbone frozen.
2. Pretrain listen and speak on unlabelled and interleaved synthetic data, with a high text fraction.
3. Train the listen head on streaming text, tags and the action channel; the speak head on text-to-speech with style.
4. Spoken SFT on tool-call traces, including fillers, truncations and re-plans.
5. DPO on text-token probabilities with timing pairs (the Moshi alignment work used 93,490 pairs and alignment over text tokens, arXiv 2506.21463).
6. RL on windows around events with per-axis timing rewards (turn onset, backchannel, yield) and voice rewards (WER, speaker similarity, quality), plus a text-quality term against drift.

Inference The staging follows the proposed recipe. Rewards come from online RL and rewards: DuplexPO decouples when to speak from what to say (arXiv 2607.07148), and Kyutai's RL on Moshi and PersonaPlex adds a response-quality reward against semantic degradation (arXiv 2606.11167). Reported The forgetting guard: speech input erodes LLM knowledge Confirmed (arXiv 2509.14930), so the downstream text LLM stays untouched by design, and the speech backbone is gated at every stage on ASR WER, strict text QA and timing.

Evaluation

• Latency and interaction. Time to first audio from gold end of turn at a fixed false-yield rate, floor-transfer offset against the human baseline, stop latency, false yields, premature takeovers, backchannel rate. Suites: Full-Duplex-Bench v1, v1.5 and v2 (arXiv 2503.04721, 2507.23159, 2510.07838) and TurnBench (arXiv 2608.25218); see conversational evaluation.
• Per component. Listen head: WER on domain audio, tag accuracy, end-of-turn recall at fixed false-positive rate. Speak head: second-ASR WER, speaker similarity, UTMOS, native-speaker panels. Tool call: does the LLM answer correctly with and without tags (an ablation of the interface). Inference
• Intelligence in speech mode. Spoken-dialogue suites (VoiceBench, arXiv 2410.17196; URO-Bench, arXiv 2502.17810) against the same LLM on text. DuplexPO's authors note that duplex gains often come with weaker reasoning (arXiv 2607.07148); here the LLM is unchanged, so I expect a small gap. Inference
• Safety. Moderate the transcript and the reply text; check voice-clone abuse and consent binding; label output (Synthesia states EU AI Act Article 50 labelling on its governance page). Inference

Serving

• Per-frame deadline. The speech LLM must finish each 80 ms frame, silent or not, and the worst frame matters, not the mean. Size it as a real-time loop; the LLM is a normal streaming text service. Inference
• Memory and sessions. A Moshi-sized backbone (temporal transformer of 32 layers and 4096 dimensions, arXiv 2410.00037) needs about 512 KiB of KV per frame, about 6.3 MiB per second and 23 GB per hour (derived, assuming full attention and bf16). Cache, not weights, sets session capacity; use grouped-query attention, a rolling window and session caps. Batch sessions to the largest $B$ with $t_{\text{step}}(B) < 80$ ms. Inference
• The hop. Keep the LLM in the same region, use prefix caching for the system prompt, route simple turns to a small model, and let fillers cover the first-token wait. Inference
• Echo. Browser AEC plus the playback reference; test stop latency after cancellation.
• Fallback. If the speech model misses deadlines or is down, route new sessions to the cascade, and hand over mid-session at a turn boundary using the transcript. Inference

Evolution to full duplex, the cascade, and risks

Inference The research track is a native full-duplex model (Moshi and PersonaPlex style: user audio, agent text, agent audio and an action token on one 80 ms clock) that replaces the two heads. The tool-call contract survives, so the downstream LLM still supplies content while the duplex model keeps the floor. The DuplexPO authors describe a tension between interactive dynamics and reasoning ability in full-duplex models (arXiv 2607.07148), which is why I keep reasoning outside it for now. The cascade of independent models stays as the baseline everything is measured against, the launch fallback, and the first generator of synthetic dialogue. It is not the target.

• Risks. Inference Tags the LLM cannot use; the hop erasing the latency gain; reward hacking (monotone prosody, an agent that never yields); scarce dual-channel data with real overlap; two models drifting apart across versions.

Baseline: a streaming ASR → LLM → TTS voice agent

This is the baseline, the launch fallback and the comparison point for the target design in the previous section, a unified speech LLM with a downstream reasoning LLM. The target replaces the independent ASR and TTS models with one speech model and keeps the LLM behind a tool call; much of this section (transport, end-of-turn, chunking, barge-in, budget) carries over to it.

A cascaded voice agent is three models plus turn logic, run as overlapping streams. The latency comes from how well the stages overlap, not from any single model. Labels: Confirmed is sourced to a paper or first-party page, Reported is a self-reported figure from the vendor's or authors' own page, Inference is design opinion.

Component pipeline

This is the cascade of the earlier chapters with turn-taking added as an engineered heuristic. The text interface between stages carries no timing, so the pipeline cannot represent "the user is still talking" without the VAD and turn manager. That is the root of the trade-offs in the last subsection.

Transport

Use WebRTC between browser or phone client and server, with a data channel for control events. LiveKit and Pipecat are the two open frameworks that package this. Inference WebRTC gives you client-side echo cancellation, jitter buffering and congestion handling for free.

Between server components, use persistent bidirectional streams. A sensible pattern is a high-volume data plane (tokens, audio) plus a separate control plane for cancel, barge-in and feedback injection. REST is a poor fit because it has no native streaming and no clean channel for cancellation. Inference

VAD and end-of-turn detection

A bare VAD with a silence timeout is the half-duplex heuristic. It adds dead air to every turn, cannot tell a pause from a finished thought, and treats overlapping speech as noise.

The usual upgrade is a VAD that triggers a semantic end-of-turn model. Pipecat's Smart Turn is an open-source model that analyses the most recent 8 seconds of the user's turn, with a recommended VAD stop time of 0.2 s. The docs say inference can run "in under 100ms" on low-cost cloud instances. Reported (Pipecat docs, vendor numbers). LiveKit Agents documents an adaptive interruption mode that uses an audio model to distinguish real interruptions from backchannel acknowledgements. Reported (LiveKit docs).

Inference Treat the end-of-turn model as a latency lever, not just an accuracy one. A fast, aggressive endpoint gives a quick reply but cuts users off mid-thought. A conservative endpoint is safe but adds dead air. A learned detector shifts that curve; it does not remove it.

Streaming ASR: partials, finals and when to start the LLM

Streaming ASR emits partials (revisable) and finals (stable). The latency/WER trade-off is set by chunk size and lookahead. Confirmed examples: SpeechLLM-XL with 1.28 s chunks reports 2.7% / 6.7% WER on LibriSpeech test clean / other (arXiv 2410.03752); Qwen3-ASR's encoder uses an attention window of 1 to 8 s so one model serves streaming and offline (arXiv 2601.21337); Kyutai DSM treats ASR as a delayed text stream over audio (arXiv 2509.08753).

LiveKit ships a version of this: preemptive generation starts the LLM when a final transcript arrives, before the turn is confirmed. The docs list it as on by default, with preemptive TTS off, long utterances skipped (10 s default), and a retry cap of 3. They warn it does not always reduce latency, so check your metrics. Reported (LiveKit docs, read 2026-10-13; check your installed version).

LLM time-to-first-token levers

LLM time-to-first-token (TTFT) is plausibly the biggest single lever in the budget Inference. The levers below are standard serving techniques:

• Prefix caching reuses KV for a shared system prompt. It needs identical token prefixes, so a prompt that changes per turn breaks the hit.
• Chunked prefill stops one long prompt from inflating tail latency for other sessions.
• Speculative decoding and quantization buy per-token speed; how hard to push is set by the budget.
• Keep concurrency limits low enough that requests queue rather than slow each other.

Inference TTFT depends on load. Inworld reports about 27,000 aggregate tokens per second and a p50 TTFT around 1.7 s for Gemma 4 31B Dense on a B200 (Inworld guide, first-party measurement) Reported. A p50 of 1.7 s at that throughput is far above a voice-agent target, so tune for latency instead: a small model or router for simple turns, and a short reply style.

LLM text to TTS: chunking

Inference The default is sentence-level chunking: buffer LLM tokens until a sentence forms, then hand it to TTS. A sentence carries a full intonation contour, so it is a safe unit. Smaller chunks cut time to first audio but risk fragmenting prosody across the boundary. Make the first chunk small and cut at the first clause boundary (comma, dash, short sentence), with a minimum length so TTS has enough text to set intonation. Make later chunks full sentences; by then the first chunk's audio is hiding the wait.

FlashTTS replaces buffer-then-emit with a lagged multi-track autoregressive design for native streaming text input (arXiv 2606.09141) Reported.

Streaming TTS first packet

Pick TTS on first-audio latency at your real input lengths and concurrency. Reported first-packet figures are self-reports with different definitions, so do not compare across rows: Moshi 200 ms in practice, 160 ms in theory (arXiv 2410.00037), Qwen3-TTS 12 Hz 97 ms (arXiv 2601.15621), Kyutai DSM-TTS 150 ms at batch size 1 (arXiv 2509.08753), FlashTTS 325 ms first-packet latency (arXiv 2606.09141), Hume Octave about 200 ms in Instant Mode (Hume docs). Reported

Measure time to first byte, real-time factor, and p95/p99 under concurrent load, because a single median hides the tail Inference. Overlap stages inside the TTS server too. Inworld and Modular report that a streaming-aware scheduler returned the first 2 seconds of synthesized audio about 70% faster than a vanilla vLLM implementation (Modular case study). Reported

Barge-in

When VAD or the turn model detects user speech during agent speech, run these steps in order:

1. Cancel the LLM stream and any queued TTS requests.
2. Flush audio buffers on both server and client playout.
3. Truncate the assistant turn in context to what was actually played, not what was generated.
4. Return to listening and let ASR take the new input.

Step 3 matters because TTS runs faster than real time; the model "said" far more than the user heard. OpenAI's Realtime API exposes this as conversation.item.truncate with an audio_end_ms offset; it cuts the audio and removes the transcript for the unplayed portion Reported (OpenAI Realtime guide). Inference In your own stack, timestamp every audio chunk sent and track the client's playout position, then cut the text at the word boundary nearest that position.

Echo cancellation is the precondition. Without it the agent hears itself and interrupts itself. The W3C Media Capture and Streams spec defines an echoCancellation property that removes sound played by the speakers from the microphone recording Confirmed (W3C spec). Inference On telephony or server-side audio you must supply AEC yourself, or gate barge-in on a higher speech threshold while the agent talks. LiveKit adds false-interruption recovery: if no transcript follows within a window (2.0 s default), the agent can resume. Reported (LiveKit docs).

Stage-by-stage latency budget

Measure from end of user speech to first agent audio at the client. Sourced figures are sparse; the target column is my illustrative target, not a measurement and not from any source.

Failure modes and mitigations

How to evaluate

• Latency: per-stage and end-to-end p50/p95/p99 under production concurrency. Track TTFB for TTS and real-time factor.
• ASR: WER on your own domain audio (names, numbers, accents), not only a public leaderboard.
• TTS: an automatic MOS predictor (UTMOS) and resynthesis WER on every change, with human CMOS panels as periodic gates.
• Interruption handling: stop latency after barge-in, false-interruption rate, and backchannel handling, scored with Full-Duplex-Bench v1 (arXiv 2503.04721) and v1.5 (arXiv 2507.23159). Those benchmarks cover pause handling, backchannel, turn-taking, interruption and overlap scenarios. Confirmed
• Online: time-to-first-audio next to an engagement metric, since a fluent reply that arrives late reads as broken.

The cascade against the target design

A cascade is the right baseline and fallback: each stage is swappable, data is abundant and errors are attributable. Its structural limits are stacked latency, information loss at each text boundary, no recovery from ASR errors, and no natural interruption or backchannel handling Inference. Native full-duplex models (Moshi, dGSLM, arXiv 2203.16502) learn turn-taking from data, and state-head designs like Freeze-Omni (arXiv 2411.00774) and MinMo (arXiv 2501.06282) add a small decision module beside the LLM instead of using external heuristics. Confirmed

The cost of going native is data: real dual-channel conversation with genuine overlap is scarce (Fisher is about 2,000 h, arXiv 2410.00037), and the DuplexPO authors describe a tension between interactive dynamics and reasoning in full-duplex models (arXiv 2607.07148). Reported Inference The target keeps the cascade's separation of content from audio but folds ASR, turn-taking and TTS into one speech model, so the turn logic is learned, not engineered. The cascade stays as the launch fallback, the comparison point and a generator of synthetic dialogue.

First-party pages cited

• Pipecat Smart Turn overview
• LiveKit turn-taking tuning
• OpenAI Realtime conversations guide
• W3C Media Capture and Streams
• Tavus CVI launch post
• Hume TTS overview
• Inworld inference guide and Modular case study

ML system design: a real-time avatar that talks back

Requirements

The user speaks, the avatar listens with visible motion, replies in a cloned voice with matched lip-sync, and can be interrupted. Synthesia's Video Agents "talk, listen, and act in real time" (Synthesia 3.0 post), while the Express-2 post makes no real-time claim and gives no latency figures. Confirmed

Latency budget

The cascade budget is the stage table in the previous section, with a render stage added. Indicative sources: the Moshi authors report 200 ms latency in practice (arXiv 2410.00037), Tavus claims less than one second between utterances (Tavus blog), and Synthesia says real dialogue is "typically under one second" (Synthesia post). Reported The targets in the table are mine, not measurements, and have no source.

Two ways to drive the renderer

Inference The comparison below is my judgement.

Inference Make A the interactive path: the speech LLM drives the streaming renderer, with fillers covering the LLM wait. Keep B as the launch fallback and the baseline on the same benchmark; route a session to B if A is down or misses its gates. Native full-duplex is the research track beyond A.

Data and training stages for A

Inference The full recipe is in the unified speech LLM section; the staging below follows OmniFlatten (modality alignment, half-duplex dialogue learning, full-duplex dialogue learning, arXiv 2410.17799):

1. Pretrain on large unlabelled audio plus transcripts (scale references: Kimi-Audio over 13M h, arXiv 2504.18425; MinMo 1.4M h, arXiv 2501.06282).
2. Modality alignment, then half-duplex dialogue, then full-duplex alignment on dual-channel data (Fisher, about 2,000 h, arXiv 2410.00037, plus synthetic TTS dialogue).
3. Post-train with WER, speaker-similarity and emotion rewards, plus preference data for expressiveness. Gate on Full-Duplex-Bench v1 and v1.5 plus human tests.

Renderer and A/V sync

A sketch; the figures below are the papers' own reports on their own hardware and protocols.

• Streaming avatar generators distil a bidirectional teacher into a causal few-step student. CausVid: 1.3 s initial latency, about 9.4 FPS (arXiv 2412.07772). Self Forcing: 17 FPS, sub-second latency, one H100 (arXiv 2506.08009). AvatarForcing: 34 ms per frame, 1.3B student (arXiv 2603.14331). SoulX-FlashTalk reports that converting the student to a strictly unidirectional architecture disrupts spatiotemporal correlations and yields rigid full-body dynamics (arXiv 2512.23379).
• Express-Render outputs 1080p at 30 fps and is distilled to two diffusion steps Confirmed (Express-2 post). A streaming student is the natural extension Inference.
• Audio is the clock. Timestamp audio and video chunks, use a client jitter buffer, and drop video frames before stalling audio. In the speech LLM path, audio frames are already on the 80 ms clock, so video blocks align to whole frames; in the fallback cascade, use TTS duration mode to match video block length.
• Play idle listening motion while the user speaks.

Barge-in and serving

In A, the speech LLM decides to yield, stops its audio, and the same decision flushes the render queue and sends the played and heard text to the text LLM. In B, barge-in follows the steps in the previous section, with the render queue added to the cancel list. Inference

For serving, use WebRTC transport (LiveKit or Pipecat) with a control data channel. Run a speech LLM pool sized for its per-frame deadline, a text LLM pool with prefix caching, and a render pool. Keep cascade pools only as failover capacity. Batch render across sessions only if the per-session budget allows. Inference

ML system design: an audio-visual avatar platform (Synthesia-style)

Requirements

One platform serves two products with different service levels. Batch video turns a script into a 1080p, 30 fps video in minutes, and dubs it into 30+ languages. Video Agents hold two-way conversations; Synthesia describes real dialogue as "typically under one second".

I would ask for cost per minute and rate limits in the interview. For comparison, Tavus claims less than one second between utterances for its conversational video interface (Tavus blog, a vendor's own claim). Reported

High-level architecture