System design

ML system design: a unified speech LLM with a downstream reasoning LLM (the target)

Why a unified speech LLM, and why a separate reasoning LLM

Inference The honest costs of the target: an extra hop of LLM time to first token; paralinguistic information must be flattened into the call, so what the tags omit the LLM never sees; and two models to serve, version and evaluate together.

Precedents for each piece. Kyutai DSM applies one architecture to both jobs: ASR delays the text behind the audio (2.5 s default), and TTS delays the audio behind the text (1.28 s), with the delay set per stream Confirmed (arXiv 2509.08753). The DSM authors report a 150 ms first-chunk latency for TTS at batch size 1 Reported. Step-Audio 2 calls external tools such as web search against hallucination and audio search to switch timbres Confirmed (arXiv 2507.16632). Qwen3-Omni uses a Thinker-Talker design, the in-model analogue of this split (arXiv 2509.17765). Freeze-Omni keeps the LLM frozen and adds a state classifier (keep listening, interrupted, turn ended) (arXiv 2411.00774). MinMo adds an 18M-parameter Full Duplex Predictor beside the LLM (arXiv 2501.06282). Both show a small decision module beside a text LLM works. Moshi's Inner Monologue shows text and audio can share one clock (arXiv 2410.00037). Confirmed

Requirements

Architecture

• Listen head. A causal streaming encoder at 12.5 Hz feeds a text-LLM-initialised backbone. The head emits streaming text (partials and stable finals), paralinguistic tags (emotion, rate, language, confidence) and an action token per frame. The user stream is ingested continuously, including while the agent speaks.
• Tool call. On an end-of-turn decision the head emits a structured call: the stable transcript, the tags, the turn state (end-of-turn confidence, whether it interrupted a reply) and a pointer to what was actually played last. Start on the end-of-turn candidate and cancel on a resumed turn, as in the cascade baseline.
• Downstream LLM. Any text LLM with tools and retrieval. It streams text back with style cues (emotion, pace) the speak head can use. It never touches audio, so it can be swapped or upgraded without retraining the speech model.
• Speak head. It turns text chunks plus style and the voice prompt into codec tokens at 12.5 Hz (semantic then acoustic codebooks through a depth transformer or multi-token head), decoded by a causal codec decoder. A small first chunk cuts time to first audio; later chunks carry more lookahead for prosody.
• Latency cover. While the call runs, the speak head says a short acknowledgement or filler, and the listen head can backchannel during long user turns. Neither waits for the LLM.
• One clock. Every output chunk carries an absolute timestamp (Qwen3-Omni assigns one temporal ID per 80 ms, arXiv 2509.17765). Confirmed The renderer and client play audio on that clock and shed video frames first.

Conversational behaviours: who decides what

Inference The speech LLM owns timing and the floor; the text LLM owns content. The action channel is one token per frame or chunk: listen, end-of-turn, backchannel, speak, yield. DuplexSLA is the precedent for such a channel (a textual action channel on a shared 160 ms chunk clock; 98.33% backchannel accuracy, self-reported; arXiv 2605.20755), and Freeze-Omni and MinMo show the same decision made by a small module beside a text LLM. Reported

Definitions and operating points are in conversational evaluation; the behaviour list follows backchannels, barge-in and self-voice. The human anchor is that listeners begin speaking a median 151 ms before the other turn ends (TurnBench, arXiv 2608.25218), and a decision at a chunk boundary waits $c/2$ on average, so decide per 80 ms frame or short chunk. Confirmed (anchor) Inference (the design reading)

Key design decisions

Data

• Listen head. Large unlabelled audio with pseudo-transcripts (Moshi uses 7M h, arXiv 2410.00037; Kimi-Audio over 13M h, arXiv 2504.18425), plus ASR data to align the encoder. Confirmed Paralinguistic tags come from pseudo-labellers (Synthesia names emotion2vec in its Express-Voice evaluation), checked on a human-labelled subset. Inference
• Speak head. TTS-grade, speaker-diverse, consented speech with style labels; interleaved synthetic speech-text (1T tokens, 600B synthetic interleaved; spoken QA rose from 13% to 31% in arXiv 2411.17607). Confirmed
• Action channel. Dual-channel conversation with real overlap: Fisher, about 2,000 h of phone conversations (arXiv 2410.00037), MinMo's 4,000 h of duplex data (arXiv 2501.06282), and Seamless Interaction conversations (used by Kyutai for RL, arXiv 2606.11167). Confirmed Synthetic dialogue has few real overlaps and regular backchannels, so mix real and synthetic.
• Tool-call traces. An LLM writes dialogues with interruptions; a harness renders both sides with many TTS voices and logs the call, the reply and the truncation. Inference The baseline cascade is the generator and the first labeller.
• Consent. Every voice clip carries a consent record and licence scope; withdrawn voices are removable from retrains. Inference

Training stages and rewards

1. Align the encoder and adaptor on ASR with the LLM backbone frozen.
2. Pretrain listen and speak on unlabelled and interleaved synthetic data, with a high text fraction.
3. Train the listen head on streaming text, tags and the action channel; the speak head on text-to-speech with style.
4. Spoken SFT on tool-call traces, including fillers, truncations and re-plans.
5. DPO on text-token probabilities with timing pairs (the Moshi alignment work used 93,490 pairs and alignment over text tokens, arXiv 2506.21463).
6. RL on windows around events with per-axis timing rewards (turn onset, backchannel, yield) and voice rewards (WER, speaker similarity, quality), plus a text-quality term against drift.

Inference The staging follows the proposed recipe. Rewards come from online RL and rewards: DuplexPO decouples when to speak from what to say (arXiv 2607.07148), and Kyutai's RL on Moshi and PersonaPlex adds a response-quality reward against semantic degradation (arXiv 2606.11167). Reported The forgetting guard: speech input erodes LLM knowledge Confirmed (arXiv 2509.14930), so the downstream text LLM stays untouched by design, and the speech backbone is gated at every stage on ASR WER, strict text QA and timing.

Evaluation

• Latency and interaction. Time to first audio from gold end of turn at a fixed false-yield rate, floor-transfer offset against the human baseline, stop latency, false yields, premature takeovers, backchannel rate. Suites: Full-Duplex-Bench v1, v1.5 and v2 (arXiv 2503.04721, 2507.23159, 2510.07838) and TurnBench (arXiv 2608.25218); see conversational evaluation.
• Per component. Listen head: WER on domain audio, tag accuracy, end-of-turn recall at fixed false-positive rate. Speak head: second-ASR WER, speaker similarity, UTMOS, native-speaker panels. Tool call: does the LLM answer correctly with and without tags (an ablation of the interface). Inference
• Intelligence in speech mode. Spoken-dialogue suites (VoiceBench, arXiv 2410.17196; URO-Bench, arXiv 2502.17810) against the same LLM on text. DuplexPO's authors note that duplex gains often come with weaker reasoning (arXiv 2607.07148); here the LLM is unchanged, so I expect a small gap. Inference
• Safety. Moderate the transcript and the reply text; check voice-clone abuse and consent binding; label output (Synthesia states EU AI Act Article 50 labelling on its governance page). Inference

Serving

• Per-frame deadline. The speech LLM must finish each 80 ms frame, silent or not, and the worst frame matters, not the mean. Size it as a real-time loop; the LLM is a normal streaming text service. Inference
• Memory and sessions. A Moshi-sized backbone (temporal transformer of 32 layers and 4096 dimensions, arXiv 2410.00037) needs about 512 KiB of KV per frame, about 6.3 MiB per second and 23 GB per hour (derived, assuming full attention and bf16). Cache, not weights, sets session capacity; use grouped-query attention, a rolling window and session caps. Batch sessions to the largest $B$ with $t_{\text{step}}(B) < 80$ ms. Inference
• The hop. Keep the LLM in the same region, use prefix caching for the system prompt, route simple turns to a small model, and let fillers cover the first-token wait. Inference
• Echo. Browser AEC plus the playback reference; test stop latency after cancellation.
• Fallback. If the speech model misses deadlines or is down, route new sessions to the cascade, and hand over mid-session at a turn boundary using the transcript. Inference

Evolution to full duplex, the cascade, and risks

Inference The research track is a native full-duplex model (Moshi and PersonaPlex style: user audio, agent text, agent audio and an action token on one 80 ms clock) that replaces the two heads. The tool-call contract survives, so the downstream LLM still supplies content while the duplex model keeps the floor. The DuplexPO authors describe a tension between interactive dynamics and reasoning ability in full-duplex models (arXiv 2607.07148), which is why I keep reasoning outside it for now. The cascade of independent models stays as the baseline everything is measured against, the launch fallback, and the first generator of synthetic dialogue. It is not the target.

• Risks. Inference Tags the LLM cannot use; the hop erasing the latency gain; reward hacking (monotone prosody, an agent that never yields); scarce dual-channel data with real overlap; two models drifting apart across versions.

Baseline: a streaming ASR → LLM → TTS voice agent

This is the baseline, the launch fallback and the comparison point for the target design in the previous section, a unified speech LLM with a downstream reasoning LLM. The target replaces the independent ASR and TTS models with one speech model and keeps the LLM behind a tool call; much of this section (transport, end-of-turn, chunking, barge-in, budget) carries over to it.

A cascaded voice agent is three models plus turn logic, run as overlapping streams. The latency comes from how well the stages overlap, not from any single model. Labels: Confirmed is sourced to a paper or first-party page, Reported is a self-reported figure from the vendor's or authors' own page, Inference is design opinion.

Component pipeline

This is the cascade of the earlier chapters with turn-taking added as an engineered heuristic. The text interface between stages carries no timing, so the pipeline cannot represent "the user is still talking" without the VAD and turn manager. That is the root of the trade-offs in the last subsection.

Transport

Use WebRTC between browser or phone client and server, with a data channel for control events. LiveKit and Pipecat are the two open frameworks that package this. Inference WebRTC gives you client-side echo cancellation, jitter buffering and congestion handling for free.

Between server components, use persistent bidirectional streams. A sensible pattern is a high-volume data plane (tokens, audio) plus a separate control plane for cancel, barge-in and feedback injection. REST is a poor fit because it has no native streaming and no clean channel for cancellation. Inference

VAD and end-of-turn detection

A bare VAD with a silence timeout is the half-duplex heuristic. It adds dead air to every turn, cannot tell a pause from a finished thought, and treats overlapping speech as noise.

The usual upgrade is a VAD that triggers a semantic end-of-turn model. Pipecat's Smart Turn is an open-source model that analyses the most recent 8 seconds of the user's turn, with a recommended VAD stop time of 0.2 s. The docs say inference can run "in under 100ms" on low-cost cloud instances. Reported (Pipecat docs, vendor numbers). LiveKit Agents documents an adaptive interruption mode that uses an audio model to distinguish real interruptions from backchannel acknowledgements. Reported (LiveKit docs).

Inference Treat the end-of-turn model as a latency lever, not just an accuracy one. A fast, aggressive endpoint gives a quick reply but cuts users off mid-thought. A conservative endpoint is safe but adds dead air. A learned detector shifts that curve; it does not remove it.

Streaming ASR: partials, finals and when to start the LLM

Streaming ASR emits partials (revisable) and finals (stable). The latency/WER trade-off is set by chunk size and lookahead. Confirmed examples: SpeechLLM-XL with 1.28 s chunks reports 2.7% / 6.7% WER on LibriSpeech test clean / other (arXiv 2410.03752); Qwen3-ASR's encoder uses an attention window of 1 to 8 s so one model serves streaming and offline (arXiv 2601.21337); Kyutai DSM treats ASR as a delayed text stream over audio (arXiv 2509.08753).

LiveKit ships a version of this: preemptive generation starts the LLM when a final transcript arrives, before the turn is confirmed. The docs list it as on by default, with preemptive TTS off, long utterances skipped (10 s default), and a retry cap of 3. They warn it does not always reduce latency, so check your metrics. Reported (LiveKit docs, read 2026-10-13; check your installed version).

LLM time-to-first-token levers

LLM time-to-first-token (TTFT) is plausibly the biggest single lever in the budget Inference. The levers below are standard serving techniques:

• Prefix caching reuses KV for a shared system prompt. It needs identical token prefixes, so a prompt that changes per turn breaks the hit.
• Chunked prefill stops one long prompt from inflating tail latency for other sessions.
• Speculative decoding and quantization buy per-token speed; how hard to push is set by the budget.
• Keep concurrency limits low enough that requests queue rather than slow each other.

Inference TTFT depends on load. Inworld reports about 27,000 aggregate tokens per second and a p50 TTFT around 1.7 s for Gemma 4 31B Dense on a B200 (Inworld guide, first-party measurement) Reported. A p50 of 1.7 s at that throughput is far above a voice-agent target, so tune for latency instead: a small model or router for simple turns, and a short reply style.

LLM text to TTS: chunking

Inference The default is sentence-level chunking: buffer LLM tokens until a sentence forms, then hand it to TTS. A sentence carries a full intonation contour, so it is a safe unit. Smaller chunks cut time to first audio but risk fragmenting prosody across the boundary. Make the first chunk small and cut at the first clause boundary (comma, dash, short sentence), with a minimum length so TTS has enough text to set intonation. Make later chunks full sentences; by then the first chunk's audio is hiding the wait.

FlashTTS replaces buffer-then-emit with a lagged multi-track autoregressive design for native streaming text input (arXiv 2606.09141) Reported.

Streaming TTS first packet

Pick TTS on first-audio latency at your real input lengths and concurrency. Reported first-packet figures are self-reports with different definitions, so do not compare across rows: Moshi 200 ms in practice, 160 ms in theory (arXiv 2410.00037), Qwen3-TTS 12 Hz 97 ms (arXiv 2601.15621), Kyutai DSM-TTS 150 ms at batch size 1 (arXiv 2509.08753), FlashTTS 325 ms first-packet latency (arXiv 2606.09141), Hume Octave about 200 ms in Instant Mode (Hume docs). Reported

Measure time to first byte, real-time factor, and p95/p99 under concurrent load, because a single median hides the tail Inference. Overlap stages inside the TTS server too. Inworld and Modular report that a streaming-aware scheduler returned the first 2 seconds of synthesized audio about 70% faster than a vanilla vLLM implementation (Modular case study). Reported

Barge-in

When VAD or the turn model detects user speech during agent speech, run these steps in order:

1. Cancel the LLM stream and any queued TTS requests.
2. Flush audio buffers on both server and client playout.
3. Truncate the assistant turn in context to what was actually played, not what was generated.
4. Return to listening and let ASR take the new input.

Step 3 matters because TTS runs faster than real time; the model "said" far more than the user heard. OpenAI's Realtime API exposes this as conversation.item.truncate with an audio_end_ms offset; it cuts the audio and removes the transcript for the unplayed portion Reported (OpenAI Realtime guide). Inference In your own stack, timestamp every audio chunk sent and track the client's playout position, then cut the text at the word boundary nearest that position.

Echo cancellation is the precondition. Without it the agent hears itself and interrupts itself. The W3C Media Capture and Streams spec defines an echoCancellation property that removes sound played by the speakers from the microphone recording Confirmed (W3C spec). Inference On telephony or server-side audio you must supply AEC yourself, or gate barge-in on a higher speech threshold while the agent talks. LiveKit adds false-interruption recovery: if no transcript follows within a window (2.0 s default), the agent can resume. Reported (LiveKit docs).

Stage-by-stage latency budget

Measure from end of user speech to first agent audio at the client. Sourced figures are sparse; the target column is my illustrative target, not a measurement and not from any source.

Failure modes and mitigations

How to evaluate

• Latency: per-stage and end-to-end p50/p95/p99 under production concurrency. Track TTFB for TTS and real-time factor.
• ASR: WER on your own domain audio (names, numbers, accents), not only a public leaderboard.
• TTS: an automatic MOS predictor (UTMOS) and resynthesis WER on every change, with human CMOS panels as periodic gates.
• Interruption handling: stop latency after barge-in, false-interruption rate, and backchannel handling, scored with Full-Duplex-Bench v1 (arXiv 2503.04721) and v1.5 (arXiv 2507.23159). Those benchmarks cover pause handling, backchannel, turn-taking, interruption and overlap scenarios. Confirmed
• Online: time-to-first-audio next to an engagement metric, since a fluent reply that arrives late reads as broken.

The cascade against the target design

A cascade is the right baseline and fallback: each stage is swappable, data is abundant and errors are attributable. Its structural limits are stacked latency, information loss at each text boundary, no recovery from ASR errors, and no natural interruption or backchannel handling Inference. Native full-duplex models (Moshi, dGSLM, arXiv 2203.16502) learn turn-taking from data, and state-head designs like Freeze-Omni (arXiv 2411.00774) and MinMo (arXiv 2501.06282) add a small decision module beside the LLM instead of using external heuristics. Confirmed

The cost of going native is data: real dual-channel conversation with genuine overlap is scarce (Fisher is about 2,000 h, arXiv 2410.00037), and the DuplexPO authors describe a tension between interactive dynamics and reasoning in full-duplex models (arXiv 2607.07148). Reported Inference The target keeps the cascade's separation of content from audio but folds ASR, turn-taking and TTS into one speech model, so the turn logic is learned, not engineered. The cascade stays as the launch fallback, the comparison point and a generator of synthetic dialogue.

First-party pages cited

• Pipecat Smart Turn overview
• LiveKit turn-taking tuning
• OpenAI Realtime conversations guide
• W3C Media Capture and Streams
• Tavus CVI launch post
• Hume TTS overview
• Inworld inference guide and Modular case study

ML system design: a real-time avatar that talks back

Requirements

The user speaks, the avatar listens with visible motion, replies in a cloned voice with matched lip-sync, and can be interrupted. Synthesia's Video Agents "talk, listen, and act in real time" (Synthesia 3.0 post), while the Express-2 post makes no real-time claim and gives no latency figures. Confirmed

Latency budget

The cascade budget is the stage table in the previous section, with a render stage added. Indicative sources: the Moshi authors report 200 ms latency in practice (arXiv 2410.00037), Tavus claims less than one second between utterances (Tavus blog), and Synthesia says real dialogue is "typically under one second" (Synthesia post). Reported The targets in the table are mine, not measurements, and have no source.

Two ways to drive the renderer

Inference The comparison below is my judgement.

Inference Make A the interactive path: the speech LLM drives the streaming renderer, with fillers covering the LLM wait. Keep B as the launch fallback and the baseline on the same benchmark; route a session to B if A is down or misses its gates. Native full-duplex is the research track beyond A.

Data and training stages for A

Inference The full recipe is in the unified speech LLM section; the staging below follows OmniFlatten (modality alignment, half-duplex dialogue learning, full-duplex dialogue learning, arXiv 2410.17799):

1. Pretrain on large unlabelled audio plus transcripts (scale references: Kimi-Audio over 13M h, arXiv 2504.18425; MinMo 1.4M h, arXiv 2501.06282).
2. Modality alignment, then half-duplex dialogue, then full-duplex alignment on dual-channel data (Fisher, about 2,000 h, arXiv 2410.00037, plus synthetic TTS dialogue).
3. Post-train with WER, speaker-similarity and emotion rewards, plus preference data for expressiveness. Gate on Full-Duplex-Bench v1 and v1.5 plus human tests.

Renderer and A/V sync

A sketch; the figures below are the papers' own reports on their own hardware and protocols.

• Streaming avatar generators distil a bidirectional teacher into a causal few-step student. CausVid: 1.3 s initial latency, about 9.4 FPS (arXiv 2412.07772). Self Forcing: 17 FPS, sub-second latency, one H100 (arXiv 2506.08009). AvatarForcing: 34 ms per frame, 1.3B student (arXiv 2603.14331). SoulX-FlashTalk reports that converting the student to a strictly unidirectional architecture disrupts spatiotemporal correlations and yields rigid full-body dynamics (arXiv 2512.23379).
• Express-Render outputs 1080p at 30 fps and is distilled to two diffusion steps Confirmed (Express-2 post). A streaming student is the natural extension Inference.
• Audio is the clock. Timestamp audio and video chunks, use a client jitter buffer, and drop video frames before stalling audio. In the speech LLM path, audio frames are already on the 80 ms clock, so video blocks align to whole frames; in the fallback cascade, use TTS duration mode to match video block length.
• Play idle listening motion while the user speaks.

Barge-in and serving

In A, the speech LLM decides to yield, stops its audio, and the same decision flushes the render queue and sends the played and heard text to the text LLM. In B, barge-in follows the steps in the previous section, with the render queue added to the cancel list. Inference

For serving, use WebRTC transport (LiveKit or Pipecat) with a control data channel. Run a speech LLM pool sized for its per-frame deadline, a text LLM pool with prefix caching, and a render pool. Keep cascade pools only as failover capacity. Batch render across sessions only if the per-session budget allows. Inference

ML system design: an audio-visual avatar platform (Synthesia-style)

Requirements

One platform serves two products with different service levels. Batch video turns a script into a 1080p, 30 fps video in minutes, and dubs it into 30+ languages. Video Agents hold two-way conversations; Synthesia describes real dialogue as "typically under one second".

I would ask for cost per minute and rate limits in the interview. For comparison, Tavus claims less than one second between utterances for its conversational video interface (Tavus blog, a vendor's own claim). Reported

High-level architecture

• Batch. Express-Voice is in-context cloning without fine-tuning, two transformers of about 800M parameters each conditioned on reference audio. Express-Animate makes motion, Express-Eval scores audio-motion agreement, and Express-Render synthesises frames, distilled to two diffusion steps. Confirmed Using Eval to select among motion candidates is my reading of its role. Inference
• Dubbing. The chain is ASR, MT, cloned-voice TTS, lip-sync re-render. The TTS needs duration control so the translated line fits the original shot. Inference
• Interactive. The voice is a unified speech LLM with a downstream text LLM, driving a causal renderer; the cascade is the launch fallback and baseline, and native full-duplex is the research track. Inference Perceiving the user needs a streaming encoder with a fixed-size KV cache (StreamingVLM reuses attention-sink states, a short window of recent vision tokens and a long window of recent text tokens; up to 8 FPS on one H100) (arXiv 2510.09608) or an omni model such as Qwen3-Omni (arXiv 2509.17765). Confirmed

Model choices and alternatives

The option descriptions are my summary Inference. Facts behind the table: Ovi uses twin DiT towers with blockwise cross-modal fusion (arXiv 2510.01284); CausVid distils a bidirectional model into a 4-step causal generator at about 9.4 FPS with 1.3 s initial latency (arXiv 2412.07772); Self Forcing reaches 17 FPS and sub-second latency on one H100 (arXiv 2506.08009). Real-time numbers are author-reported on their own hardware and not comparable. Confirmed Codec trade-offs are in Codecs and speech tokenizers.

One timeline for voice and video

The two teams need a shared clock, or sync becomes a post-hoc patch. Ovi scales audio RoPE frequencies by 31/157 so 31 video latent frames align with 157 audio tokens (arXiv 2510.01284). Qwen3-Omni instead assigns absolute-time IDs (one temporal ID per 80 ms of audio) and adjusts video frame IDs to real timestamps, which allows streaming inputs of arbitrary duration (arXiv 2509.17765). Confirmed Inference I would use absolute-time positions across every model, so a TTS chunk, a motion chunk and a rendered block all carry the same timestamps. The audio stays the clock, as in the real-time avatar section. A speech LLM on an 80 ms frame clock gives the voice side of that timeline for free.

Data

• Capture studio against webcam or photo. Synthesia supports personal avatars from a single photo and an optional voice sample (personal avatars docs). Confirmed Inference Studio capture gives clean, controlled full-body data for the premium tier and for training. Webcam and photo give scale and diversity but noisy lighting and audio; tier them by capture quality.
• Curation. Ovi filters clips with SyncNet (absolute offset at most 3, confidence above 1.5, mean volume at least -60 dB) and uses MLLM captions (arXiv 2510.01284). Confirmed
• Multilingual speech. Inference Dubbing needs paired speech across 30+ languages with accent labels and speaker identity; emotion2vec tags give emotion coverage. Long-tail languages need synthetic or licensed data.
• Consent provenance. Inference Every training clip carries a consent record and licence scope. A person who withdraws consent must be removable from the training set and from future retrains.

Training stages

1. Pretrain backbones. Start from an open video DiT (Wan is an open suite of video foundation models, arXiv 2503.20314; Ovi initialises its video tower from Wan2.2 5B, arXiv 2510.01284) and a speech AR model with a codec. Confirmed
2. Audio-conditioning finetune. Add audio conditioning for lip and motion; FantasyTalking adds a second, frame-level lip-refinement stage (arXiv 2504.04842). Confirmed
3. Distillation. Reduce to few steps for batch; build a causal student for interactive, with autoregressive rollouts in training to close the train-test gap (Self Forcing, arXiv 2506.08009). Confirmed
4. Preference optimisation. FantasyTalking2 trains a reward model (Talking-Critic), curates a 410K-pair preference dataset and optimises with TLPO (arXiv 2508.11255). Confirmed For voice, use the RL recipes from the control-and-quality chapter: WER and SIM rewards plus an emotion term, and add per-axis timing rewards for the speech LLM's action channel.

Evaluation

Papers report Sync-C under different protocols, so I treat cross-paper numbers as not comparable Inference. Build one internal benchmark with fixed identities, languages and scripts, run by both the Voice and video teams, and pin metric variants.

Serving and cost

• Pools. Inference Separate GPU pools for the speech LLM, text LLM, motion and render; cascade pools only as failover capacity. Interactive gets reserved capacity with a hard concurrency cap; batch fills the gaps from a queue and can be pre-empted or delayed.
• Caching. Inference Cache identity features and voice conditioning per avatar, so repeat renders skip the encoders.
• Steps against quality. Express-Render uses two diffusion steps, and Render-Turbo is described as coming later (Express-2 post). Confirmed Inference Treat step count as a per-tier knob: fewer steps for previews and interactive, more for final export.
• Autoscaling. Inference Scale batch on queue depth and interactive on session count, with warm pools because model load is slow.

Safety

• Consent gate. Synthesia requires a live consent video (uploads not accepted) and the person must match the photo (personal avatars docs). Confirmed Inference Bind the voice store to the same verified identity, so a cloned voice cannot be used without it.
• Moderation at creation. Synthesia says it moderates content at the point of creation and lists a Content Integrity Policy (AI governance page). Confirmed Inference Screen the script before generation and the output before delivery.
• Labelling and provenance. The governance page states EU AI Act Article 50 labelling and lists Content Authenticity Initiative engagement. Confirmed The page does not mention C2PA or watermarking.
• Voice-clone abuse. Synthesia states that it strictly prohibits cloning voices without explicit consent (voice cloning page). Confirmed Inference Add speaker verification of the reference against the consenting person, and rate limits on cloning and on interactive agents.

Risks and what I would do first

• Risks. Inference Audio and video teams drift on clocks and metrics; distilled causal renderers degrade under audio conditioning; action-channel data with real overlap is scarce; tags the text LLM cannot use; reward models get gamed.
• Days 1-30. Inference Learn the current Voice stack and its failures. Build the shared benchmark and metric pins with the video team, including latency percentiles, and put the cascade on it as the baseline.
• Days 31-60. Inference Stand up the unified speech LLM path: listen head with tags and end-of-turn, tool call to the text LLM, speak head with fillers, absolute-time chunks and listening cues, scored on the benchmark. Ship the cascade as launch fallback behind the same gates.
• Days 61-90. Inference Train the first in-house version on the shared timeline, add timing and voice rewards with hacking guards, gate it against the cascade, and start the native full-duplex research track.