Multimodal generation

Audio-visual diffusion

Three threads have converged. Large DiT video backbones now take audio as a condition or generate it jointly. Avatar models bolt audio conditioning and long-video stitching onto those backbones. Causal few-step distillation turns the result into a streamer.

Real-time numbers in this section are author-reported, often multi-GPU, and not comparable across papers.

Video diffusion backbones

The shared recipe is a 3D causal VAE, a DiT denoiser, and a flow-matching or rectified-flow objective Inference. Models differ in attention sparsity, text encoders, and how causal the VAE is.

Audio-driven human animation

These models take a reference image plus audio and render a talking person. Most reuse a video foundation model (Wan, Hunyuan) and add audio cross-attention Inference.

• OmniHuman-1 (arXiv 2502.01061): a DiT that scales up data by mixing motion-related conditions into training. OmniHuman-1.5 (arXiv 2508.19209): an MLLM writes a structured textual representation for semantic and emotional guidance; a "Pseudo Last Frame" design mitigates inter-modality conflicts. The authors argue earlier models sync to low-level cues such as audio rhythm and lack understanding of emotion and intent.
• HunyuanVideo-Avatar (arXiv 2505.20156): an MM-DiT for multi-character, emotion-controllable dialogue video. A face-aware audio adapter isolates each character with a latent-level face mask, so audio is injected per character through cross-attention.
• FantasyTalking (arXiv 2504.04842): audio-to-video in a pretrained video DiT, then lip-focused refinement. FantasyTalking2 (arXiv 2508.11255): preference optimisation (TLPO) with a reward model, Talking-Critic, and a 410K-pair dataset.
• MultiTalk (arXiv 2505.22647): multi-person conversation with per-stream audio; Label RoPE (L-RoPE) resolves which audio stream binds to which person. InfiniteTalk (arXiv 2508.14033): sparse-frame dubbing for infinite-length sequences, editing full-body motion rather than only the mouth region.
• EchoMimicV3 (arXiv 2507.03905): a 1.3B-parameter framework that unifies multi-task and multi-modal human animation.
• Synthesia Express-2 (4 Sep 2025): Express-Voice, Express-Animate (co-speech motion from audio), Express-Eval and Express-Render (1080p, 30 fps, arbitrary length). Express-Render is distilled to two diffusion steps. The post makes no real-time claim Reported.

Joint audio and video generation

Joint models replace post-hoc lip sync by generating both streams together. Audio quality is still the open question.

Ovi (Character.AI, arXiv 2510.01284) is the clearest open recipe.

• Twin DiT towers: video initialised from Wan2.2 5B; the audio tower has the same shape (30 blocks, dim 3072) and trains from scratch.
• Bidirectional audio-video cross-attention in every block.
• RoPE on both streams, with the audio frequencies scaled to match the coarser video resolution. Video latents span 31 frames while audio latents form 157 tokens.
• The authors note that dense blockwise fusion costs time per sampling step, and name distillation such as DMD2 as a way to reduce steps.

UniVerse-1 (arXiv 2509.06155) takes the other route. It stitches pretrained experts, Wan2.1 (1.3B) for video and ACE-Step (3.5B) for music, with lightweight cross-modal MLP connectors in corresponding blocks. It was fine-tuned on about 7,600 hours of audio-video data.

Coupling therefore comes in two patterns: same-shape twin towers with per-block cross-attention (Ovi), or pretrained experts joined by connectors (UniVerse-1). Ovi handles the token-rate mismatch with RoPE rescaling.

Streaming and real-time

The recipe is to distil a bidirectional teacher into a causal few-step student with a KV cache. The failure mode is error accumulation over long rollouts (CausVid, Self Forcing and AvatarForcing all discuss it).

Error accumulation has several named causes and fixes. The train-test gap is closed by training on own rollouts (Self Forcing). LiveTalk reports flicker and black frames when Self Forcing meets multimodal conditioning, and repairs them with a better distillation recipe. Identity drift is handled with identity sinks (LiveTalk) and a reference sink (StreamAvatar). Mechanisms are rolling or sink KV caches, on-policy rollouts and distribution matching.

Decoupled Self-Forcing (arXiv 2609.10317) frames two routes: end-to-end audio-conditioned video diffusion (high quality, but only at large scale) against two-stage methods that generate an intermediate motion representation (cheaper, lower fidelity). Its own approach pairs a small causal AR motion generator with a diffusion renderer and reports 15.4 FPS at 1.3 s latency. SoulX-FlashTalk (arXiv 2512.23379) argues that strictly unidirectional attention often compromises visual fidelity.

Metrics and data

• Lip sync: Ovi's data filter uses SyncNet to filter out clips lacking audio-video synchronization (arXiv 2510.01284); UniVerse-1 reports LSE-C (arXiv 2509.06155).
• Identity: UniVerse-1 scores identity as mean DINOv3 feature similarity between the reference image and each generated frame (arXiv 2509.06155).
• Datasets named in the papers above: HDTF, CelebV-HQ and EMTD (InfiniteTalk), and HDTF, AVSpeech and CelebV-HQ (LiveTalk).
• Human preference studies (Ovi ran a blind pairwise study with 50 participants) and reward-model judges (Talking-Critic, arXiv 2508.11255) appear in the papers above.
• Ovi's curation also filters clips on a minimum mean volume and adds captions (arXiv 2510.01284).

Autoregressive and unified models

The field has moved from a discrete-versus-continuous fight to hybrids. Next-token AR dominates unified stacks, but the per-token head is increasingly diffusion or flow over continuous latents. Video is moving to chunked, causal AR with diffusion for streaming.

Discrete-token AR

• VAR (arXiv 2404.02905): next-scale prediction. The authors report ImageNet 256 FID improving from 18.65 to 1.73 against an AR baseline, with around 20x faster inference.
• Infinity (arXiv 2412.04431): bitwise token prediction with an infinite-vocabulary tokenizer and bitwise self-correction. The authors report GenEval 0.62 to 0.73 over SD3-Medium, and a 1024 px image in 0.8 s, 2.6x faster than SD3-Medium.
• Emu3 (arXiv 2409.18869): one transformer trained from scratch with next-token prediction over discrete image, text and video tokens. The authors claim it surpasses SDXL and LLaVA-1.6.
• Emu3.5 (arXiv 2510.26583): world-model framing on interleaved vision-language data, over 10T tokens, RL post-training. Discrete Diffusion Adaptation (DiDA) turns token-by-token decoding into bidirectional parallel prediction, accelerating per-image inference by about 20x (authors).

Continuous-token AR with diffusion heads

• MAR (arXiv 2406.11838, NeurIPS 2024): no vector quantisation; each token's distribution is modelled with a diffusion procedure (Diffusion Loss), including masked autoregressive variants.
• Fluid (arXiv 2410.13863): continuous tokens give clearly better visual quality than discrete; random order beats raster on GenEval. At 10.5B: zero-shot FID 6.16 on MS-COCO 30K, GenEval 0.69.
• NOVA (arXiv 2412.14169, ICLR 2025): non-quantised AR video, frame by frame in time and set by set within a frame; 0.6B parameters.

Chunked AR video diffusion

• MAGI-1 (Sand AI, arXiv 2505.13211): autoregressively predicts fixed-length video chunks, trained to denoise per-chunk noise that increases monotonically over time, which supports streaming generation. The largest variant has 24B parameters and supports context lengths up to 4 million tokens (authors).
• Self Forcing (arXiv 2506.08009, NeurIPS 2025): trains on its own rollouts with a rolling KV cache. The authors report 17 FPS with sub-second latency on one H100. Inference It is the most relevant entry for interactive avatars.

Hybrid unified models

One related observation from the Kelix report: most VLMs pair discrete text tokens with continuous ViT features, which biases them toward understanding (arXiv 2602.09843).

Discrete diffusion language models

• LLaDA (arXiv 2502.09992): 8B masked-diffusion LM trained from scratch; the authors report it is competitive with LLaMA3 8B in in-context learning.
• MMaDA (arXiv 2505.15809): a multimodal diffusion language model with UniGRPO, a unified policy-gradient RL algorithm for post-training across reasoning and generation.

The convergence debate

• Continuous tokens with a diffusion or flow head give better per-token quality (Fluid, MAR). Discrete tokens give a simpler LM interface and a single cross-entropy (Emu3, LLaDA) Inference.
• Random-order or bidirectional-within-block generation is winning for images. Raster order is kept where causal streaming matters Inference.
• Chunked, causal AR suits streaming video (MAGI-1, Self Forcing). Full-clip diffusion remains the offline quality leader Inference.

Tokenizers beyond speech

Across images, video and audio, the tokenizer matters more than the generator. The consensus design is a causal, heavily compressed latent, regularised toward semantic features Inference. Discrete codecs stay the default for audio LMs; continuous VAE latents stay the default for diffusion and flow.

Image and video tokenizers

• MAGVIT-v2 (arXiv 2310.05737, ICLR 2024) uses lookup-free quantisation (LFQ). With this tokenizer, the authors show LLMs outperforming diffusion models on ImageNet and Kinetics, and title the paper "Tokenizer is Key to Visual Generation" Reported.
• FSQ (arXiv 2309.15505) quantises each latent dimension to a small set of fixed values. The codebook is the implicit product of those sets, so the authors report no codebook collapse and no need for commitment losses or codebook reseeding Confirmed.
• TiTok (arXiv 2406.07550) is a 1D tokenizer. It reduces a 256x256x3 image to 32 discrete tokens, with gFID 1.97 on ImageNet 256 using the same generator framework Reported.
• The NVIDIA Cosmos tokenizer outputs either continuous latents or discrete tokens for images and video. Its README states total compression of up to 2048x (8x16x16) and running up to 12x faster than the best available tokenizers Reported.
• Causal 3D VAEs compress time and space without peeking at future frames. Wan's VAE replaces GroupNorm with RMSNorm to preserve temporal causality, which enables a feature cache (arXiv 2503.20314) Confirmed.

FSQ in standard notation (not taken from the sources): bound each latent dimension, then round.

Here $L_i$ is the number of levels in dimension $i$, and $|\mathcal{C}|$ is the implicit codebook size.

Semantic-aligned VAEs

A recent trend is to align the latent with, or build it directly on, vision foundation model features, then decode.

General audio and music codecs

Unified and joint audio-visual tokenizers

• UniCodec (arXiv 2502.20067): one codec with a single codebook across speech, music and sound, using a partitioned domain-adaptive codebook and domain Mixture-of-Experts.
• MelTok (arXiv 2510.01903): a 2D mel-spectrogram tokenizer that compresses 44.1 kHz audio into a single codebook, with a token-based vocoder.
• EntangleCodec (arXiv 2606.02739): a unified discrete audio tokenizer with caption-aligned semantic-acoustic representations and a flow-matching diffusion decoder.
• AVTok (arXiv 2606.30811): a 1D joint audio-video tokenizer with a shared encoder-decoder, modality-specific learnable queries and a unified codebook Reported.
• VideoPoet (arXiv 2312.14125) is the precedent: it concatenated MAGVIT-v2 and SoundStream vocabularies into one unified vocabulary rather than learning a joint tokenizer.

Trade-offs

The table is directional, not a benchmark Inference.

Bridge to speech codecs

Speech codecs (Mimi, X-Codec2, BiCodec, CosyVoice 2 FSQ tokens) are covered in chapter 03. The same logic applies there: low rate and semantic distillation help the LM, continuous latents win back fidelity. Inference For a Voice team the new point is that audio-visual tokenizers may be converging on the same recipe as speech ones.

Multimodal LLM architecture

Open multimodal LLMs (Qwen3-VL, InternVL3.5, Llama 4) share one template: a vision encoder, a token-reducing connector, and a decoder LLM with multimodal positional encoding. The differences are in positional encoding, visual token count and MoE. Each claim below is cited to the model's technical report, its arXiv paper or Meta's own post.

Encoder, connector, LLM

• Qwen3-VL uses DeepStack to integrate multi-level ViT features, plus an enhanced interleaved MRoPE and explicit text timestamps for video. It has dense 2B/4B/8B/32B and MoE 30B-A3B and 235B-A22B variants, with native 256K context (arXiv 2511.21631) Confirmed.
• Its vision encoder is SigLIP-2, a two-layer MLP merger compresses visual features into tokens, and DeepStack routes visual tokens from several ViT layers into corresponding LLM layers through lightweight residual connections (arXiv 2511.21631) Confirmed.
• InternVL3.5 compresses 1024 visual tokens per image patch to 256 with a pixel shuffle module. Its Visual Resolution Router (ViR) dynamically adjusts the resolution of visual tokens (arXiv 2508.18265) Reported.
• Llama 4 uses a MetaCLIP-based vision encoder trained separately in conjunction with a frozen Llama, with MoE (Scout: 17B active, 16 experts, 109B total; Maverick: 17B active, 128 experts, 400B total; Meta's Llama 4 post) Reported.

Positional encoding

Video is handled as time-indexed positions. That one choice enables audio-video sync and streaming Inference.

• Qwen3-Omni's TM-RoPE splits rotary dimensions into temporal, height and width (24/20/20 angles). Audio gets one temporal ID per 80 ms. Video frame IDs are adjusted to real timestamps (arXiv 2509.17765 section 2.3) Confirmed.

Early vs late fusion

Meta describes Llama 4 as using early fusion to integrate text and vision tokens into a unified model backbone (Meta's Llama 4 post) Reported.

Omni models and time alignment

Qwen3-Omni is a Thinker-Talker MoE (arXiv 2509.17765). The AuT audio encoder runs at a 12.5 Hz token rate. The Talker generates one codec frame per step, an MTP module produces the residual codebooks, and a lightweight causal ConvNet (Code2Wav) renders the waveform Confirmed.

Qwen3-Omni's theoretical first-packet latency in cold-start settings is 234 ms for audio and 547 ms for video. At 4 concurrency, Table 2 gives 728 ms and 1517 ms Confirmed. These are theoretical figures, not end-to-end measurements.

Training and RL

• InternVL3.5 uses Cascade RL: offline RL for stable convergence, then online RL for refined alignment (arXiv 2508.18265) Reported.
• A study of R1-like training for vision-language models reports that SFT before RL can undermine RL by inducing imitated "pseudo reasoning paths". Its GRPO variant with a mixed perception and cognition reward fostered more adaptive reasoning (arXiv 2504.11468) Reported.

Streaming video understanding

StreamingVLM (arXiv 2510.09608) keeps a fixed-size KV cache of attention sinks, recent vision tokens and recent text tokens. It uses contiguous RoPE. It runs at up to 8 FPS on one H100 on a Qwen2.5-VL-7B base Confirmed. Authors report a 66.18% win rate against GPT-4o mini on Inf-Streams-Eval and +4.30 on LongVideoBench Reported. Related: ReKV (arXiv 2503.00540) stores video KV caches in RAM and disk and retrieves only the query-relevant ones.

Evaluation caveat

What an avatar that perceives the user needs

Speech-LLM design, the audio half of this stack, is in chapter 02 and chapter 04.