Speech-to-speech modelling

Control and quality

Steering voice, tone and emotion

Five control channels, usually combined: a reference clip, a natural-language instruction, inline tags, duration targets, and style guidance at sampling time.

Figure: each control channel enters the token language model as a separate condition; the decoder turns the resulting tokens into audio.

Control channels

Channel	Examples	Notes
In-context cloning	Synthesia Express-Voice	Synthesia describes a two-stage transformer with autoregressive and non-autoregressive parts (about 800M parameters each), operating on graphemes with Descript's RVQ tokenizer. It is conditioned on reference audio, with no fixed speaker embedding (Synthesia Express-2 post)
NL instructions	CosyVoice 3, Parler-TTS, IndexTTS2, Hume Octave, gpt-4o-mini-tts, Qwen-Audio-3.0-TTS	CosyVoice 3 prepends a natural-language description and an `<endofprompt>` token to the text; instruction-following data grew from 1,500 to 5,000 hours (arXiv 2505.17589). Parler-TTS reads gender, speaking rate, pitch and reverberation from the text description, with no reference audio; Parler-TTS Large is 2.3B parameters (Parler-TTS README). Hume Octave 1 takes a natural-language `description` and a separate `speed` multiplier from 0.5 to 2.0 (Hume docs). OpenAI's TTS guide shows an `instructions` field on gpt-4o-mini-tts. Qwen-Audio-3.0-TTS offers free-style instructions and inline tags (arXiv 2607.23938)
Inline tags	ElevenLabs audio tags, CosyVoice 3, Qwen-Audio-3.0-TTS	ElevenLabs lists bracket tags such as `[whispers]`; its docs say a tag for a delivery the voice was not trained on may be less reliable (ElevenLabs docs). CosyVoice 3 supports vocal bursts between text tokens and feature tags such as `[laughter]` and `[breath]` (arXiv 2505.17589)
Duration control	IndexTTS2 (arXiv 2506.21619)	Explicit token count for exact timing, or free mode that keeps prompt prosody
Style guidance	CFG on the style or text condition	Inference Reuses the CFG term from the flow-matching material in chapter 03 (Speech generation), with a style-only unconditional branch

Disentanglement

IndexTTS2 separates emotion from timbre: timbre comes from one prompt and emotional tone from a style prompt, so the two can come from different speakers (arXiv 2506.21619). Its text-based emotion control comes from fine-tuning Qwen3. The authors report better word error rate, speaker similarity and emotional fidelity than other zero-shot TTS models. Step-Audio-EditX takes a different route: large-margin synthetic data instead of representation-level disentanglement (arXiv 2511.03601).

In CosyVoice's own tests, the instruct model raised the share of generated speech that an emotion2vec classifier labelled sad, angry, surprised and disgusted, compared with the base model (arXiv 2407.05407, Table 10).

Duration control and lip-sync

The IndexTTS2 authors motivate exact durations by video dubbing, where speech must match a video block length (arXiv 2506.21619). Inference This is the hook for avatar sync.

Evaluation

emotion2vec (arXiv 2312.15185) is a self-supervised speech-emotion representation, usable as the basis for emotion similarity. Synthesia reports measuring Express-Voice with WavLM for speaker similarity and emotion2vec for emotional similarity.

Synthesia context

Synthesia's Express-2 (4 Sep 2025) pairs Express-Voice with Express-Video. The post reports a blind test with 100 native English-speaking evaluators across 17 accents, in which Express-Voice rated highest for matching the original speaker's identity, rhythm and accent (Reported company claim). The post gives no scores.

Gap

In-context cloning internals, speaker embeddings and timbre/prosody/content disentanglement were not researched beyond IndexTTS2 and Express-Voice.

In the interview, say:

"I would combine a reference clip for identity, an NL or tag channel for emotion, and a duration mode for sync, with emotion kept disentangled from timbre so style transfers across speakers."

Post-training and evaluation for speech generation

Post-training uses ASR and speaker-verification rewards. The recurring failure is reward hacking.

Figure: the RL loop for TTS. Automatic rewards score each rollout; human listening tests audit the rewards and gate release.

Methods

Method	Idea
Seed-TTS RL (arXiv 2406.02430)	RL for robustness, speaker similarity, controllability
DiffRO (CosyVoice 3; arXiv 2505.17589, 2507.05911)	Computes rewards on neural codec tokens rather than synthesized audio; Gumbel-Softmax makes the reward differentiable; a multi-task reward model adds emotion and quality control
F5R-TTS (arXiv 2504.02407)	Converts flow outputs to Gaussians so GRPO applies; WER and SIM rewards; 29.5% relative WER reduction, 4.6% relative SIM gain (authors)
Flow-GRPO (arXiv 2505.05470)	ODE-to-SDE conversion that keeps the model's marginals, enabling the sampling that online RL needs for flow-matching models (text-to-image)
GROW (arXiv 2608.03215)	Group-relative advantage-weighted on-policy RL that acts on the standard flow-matching objective, avoiding the ODE-to-SDE conversion and per-step likelihood ratios; on DiTAR the authors report WER 2.016 to 1.558 and SIM 0.676 to 0.715
SwanVoice (arXiv 2605.30993)	DiffusionNFT post-training with phone-level and speaker-similarity rewards
FlowTTS-GRPO (arXiv 2606.23190)	Online RL for flow-matching TTS: ODE-to-SDE conversion, weighted multi-objective reward; tested on CosyVoice 3.0 and F5-TTS
Step-Audio-EditX (arXiv 2511.03601)	Large-margin synthetic data for iterative emotion and style editing, with no embedding priors or auxiliary modules
Preference optimisation	EASPO: stepwise preference optimisation for diffusion TTS, scoring noisy intermediate states (arXiv 2509.25416)

Reward hacking

Trap: automatic rewards get gamed

RRPO reports that DiffRO-style reward models can be exploited with acoustic artifacts that earn spurious reward but degrade perceptual quality (arXiv 2512.04552).
A prosody study found GRPO on CER and NLL rewards lowered error rates but collapsed prosody into monotone speech; adding speaker similarity destabilised training. Its fix was iterative DPO with a few hundred human preference pairs per round (arXiv 2509.18531).
FlowSE-GRPO (speech enhancement): a single reward gives fast metric gains but degrades audio fidelity; multi-metric rewards mitigate this (arXiv 2601.16483).

Inference

Inference Always pair automatic rewards with a naturalness term and listening tests.

Metrics

Metric	Notes
WER	Seed-TTS-eval uses Whisper-large-v3 (English) and Paraformer-zh (Mandarin)
SIM	Seed-TTS-eval uses WavLM-large fine-tuned for speaker verification. Inference Other benchmarks may use other verification models, so quote the variant
UTMOS	Predicted MOS used as a naturalness check alongside WER and SIM (for example in GROW, arXiv 2608.03215)
emotion2vec	Emotion similarity (arXiv 2312.15185)
Full-Duplex-Bench (arXiv 2503.04721)	Pause handling, backchanneling, turn-taking and interruption management for spoken dialogue models
Human listening	Needed because the automatic metrics are optimisable

Link to my work

The GRPO experience from online RL at Amazon transfers directly. The new part is making rollouts differentiable or likelihood-tractable for flow models.

In the interview, say:

"I would use WER and SIM rewards plus an emotion or naturalness term, watch for monotone prosody as a hacking signature, and gate releases on human listening tests."