MiniMaxAI/MiniMax-H3

Overview

MiniMax H3 is an open-weight, general-purpose multimodal generation model. Rather than being confined to one specialized task — generate, edit, or reference — H3 reads a multimodal context that mixes text, images, video, and audio together, interprets the creative intent as a whole, and produces coherent audio-visual output end to end.

Architecturally it is a CFG-distilled joint video/audio diffusion transformer served through vLLM-Omni's OpenAI-compatible /v1/videos API. Every request returns a single MP4 containing H.264 video and native stereo audio — the audio is generated jointly with the video by the same DiT, not dubbed on afterwards.

Three capability areas the model targets:

• Commercial-grade generation — film and entertainment, advertising and branding, e-commerce, gaming; dynamic typography, VFX, product showcases, UI/UX motion design.
• Native multimodal understanding + generation — interprets characters, motion, sound, emotion, cinematography, and visual style across mixed references, then combines them.
• Precise multimodal editing and control — iterative refinement of characters, objects, scenes, sound, and rhythm with strong instruction following.

The checkpoint ships as two independently served partitions:

Partition	Tasks	Conditioning
FL2VA	t2va, fl2va	text only, or text + first/last frame
Ref2VA	ref2va	text + omni references (images, video clips, audio clips)

One server process loads one partition, so switching between T2VA/FL2VA and Ref2VA means restarting the server against the other one.

Current vLLM-Omni main also supports a combined root service through #5720: serving /path/to/MiniMax-H3 loads both DiTs while sharing the encoder and VAEs. The partition-specific commands below remain the qualified memory-conscious path.

Components (BF16): a 52-block joint video/audio DiT (66.3 GB), a Qwen3-VL layer-50 text/vision encoder (51.5 GB), a video VAE (~10 GB) and an audio VAE (~0.6 GB).

Model specifications

Output duration	4–15 seconds
Frame rate	24 FPS (fixed)
Audio	Native stereo, on every generation
Resolution — 1440p (2K)	Short edge 1440 px for 16:9 … 9:16; wider formats ≈3.7 MP (e.g. 2976×1248 at 21:9). (mode coming soon)
Resolution — 768p	Short edge 768 px for 16:9 … 9:16; wider formats ≈1 MP (e.g. 1536×672 at 21:9). Upscalable to 1440p.
Aspect ratio	First/Last Frame follows the uploaded image. Text-to-Video and Omni Reference select from 21:9, 16:9, 4:3, 1:1, 3:4, 9:16 — Omni Reference also offers Auto
Prompt length	Up to 7,000 characters

Input modes and limits

First/Last Frame — 0, 1, or 2 images. Edge lengths in [256, 5760], aspect ratio 5:2 to 2:5. With no image, H3 falls back to Text-to-Video.

Omni Reference — up to 9 images; up to 3 video clips (2–15 s each, 15 s total); up to 3 audio clips (2–15 s each, 15 s total). Edge lengths in [256, 5760], video aspect ratio 5:2 to 2:5. Audio must be paired with at least one image or video — it cannot be the sole reference. Mixed inputs cap at 12 files total; with none, H3 falls back to Text-to-Video.

	Formats	Per-file size
Video	H.264/AVC, H.265/HEVC (embedded audio AAC or MP3)	50 MB
Image	JPG, JPEG, PNG, WEBP, HEIC, HEIF	30 MB
Audio	WAV, MP3	15 MB

Limits are per file, not per combined upload. Keep the whole request body under 64 MB — prefer URL-based media over inlined uploads for anything large.

Prerequisites

H3 is served from a local directory, not the HF id — the checkpoint's two partitions are subdirectories, and one server process loads one of them. Request access on the model card (the repo is gated), then download it:

hf download MiniMaxAI/MiniMax-H3 --local-dir /path/to/MiniMax-H3

That yields /path/to/MiniMax-H3/FL2VA and /path/to/MiniMax-H3/Ref2VA. Substitute your own --local-dir throughout the commands below.

ffmpeg and ffprobe must be on PATH — they are used for reference-video preparation and MP4 muxing.

Installation

Docker (recommended)

The image bundles the H3 handlers and the FlashAttention-4 kernels, so there is nothing else to install:

docker pull vllm/vllm-omni:minimax-h3

Because H3 is served from a local directory, the container needs that directory bound at the same absolute path — the command builder's Docker mode emits the bind mount for you:

-v /path/to/MiniMax-H3/FL2VA:/path/to/MiniMax-H3/FL2VA:ro

pip

H3 support ships in vLLM-Omni rather than the vllm wheel, so the pip path needs a source checkout:

uv venv
source .venv/bin/activate
uv pip install vllm==0.26.0
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e '.[fa4]'

The [fa4] extra installs the CuTe-DSL FlashAttention-4 kernels — CUDA-only, and used on Blackwell. Drop the extra and the FLASH_ATTN backend falls back to FA3/FA2.

Launch — two 24/32 GB GPUs with distributed layerwise offload

vLLM-Omni includes a low-memory TP2 path for H3. Install the current vLLM-Omni source checkout as shown above.

The two-GPU profile combines TP2 with distributed layerwise offload (DLO). Each rank keeps its rank-local TP shard in pinned host memory; --dlo-no-use-allgather streams those shards directly instead of reconstructing full DiT blocks on every rank. A configurable prefix of DiT blocks stays resident for the complete denoise stage, then releases its HBM before VAE decode.

Select the profile first. RTX 5090 uses 20 resident DiT layers at the released 1344×768 shape; RTX 4090 keeps more HBM headroom with 12 resident layers and a 1024×576 starting shape:

# 2× RTX 5090 (32 GB)
export DLO_RESIDENT_LAYERS=20 WIDTH=1344 HEIGHT=768

# Or, 2× RTX 4090 (24 GB)
# export DLO_RESIDENT_LAYERS=12 WIDTH=1024 HEIGHT=576

Start the FL2VA partition for T2VA and FL2VA:

CUDA_VISIBLE_DEVICES=0,1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \
vllm serve /path/to/MiniMax-H3/FL2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --num-gpus 2 \
  --tensor-parallel-size 2 \
  --usp 1 \
  --ring 1 \
  --text-encoder-tp-size 2 \
  --vae-patch-parallel-size 2 \
  --vae-parallel-mode tile \
  --vae-use-tiling \
  --enable-distributed-layerwise-offload \
  --dlo-no-use-allgather \
  --dlo-resident-layers "${DLO_RESIDENT_LAYERS}" \
  --enforce-eager \
  --diffusion-attention-backend CUDNN_ATTN

The resident count changes placement and transfer frequency only; it does not quantize or change the BF16/FP32 denoise math. Use a 384 GiB-class host for the pinned model shards and offload buffers.

Wait for Application startup complete, then verify health and run one full 50-step T2VA request:

curl --fail http://127.0.0.1:8000/health

curl --fail-with-body -sS \
  -X POST http://127.0.0.1:8000/v1/videos/sync \
  -F 'prompt=At night, three cats march into a bedroom playing tiny brass instruments, then abruptly file out, with synchronized room ambience.' \
  -F "width=${WIDTH}" \
  -F "height=${HEIGHT}" \
  -F 'fps=24' \
  -F 'num_inference_steps=50' \
  -F 'flow_shift=12' \
  -F 'seed=1101' \
  -F 'extra_params={"task":"t2va","duration":5.0,"audio_flow_shift":3.0}' \
  -o minimax-h3-t2va.mp4

ffprobe -v error -show_entries \
  stream=index,codec_name,width,height,r_frame_rate,sample_rate,channels \
  -of json minimax-h3-t2va.mp4
ffmpeg -v error -i minimax-h3-t2va.mp4 \
  -map 0:v:0 -map 0:a:0 -f null -

A successful output has H.264 video at 24 FPS and stereo AAC audio at 32 kHz, and the final ffmpeg command exits with status 0. One server loads one checkpoint partition; stop it and restart the same command with /path/to/MiniMax-H3/Ref2VA before Ref2VA requests.

The PR includes an all-task runner that starts both partitions in sequence, exercises T2VA, FL2VA, image+audio Ref2VA, and two-video Ref2VA, validates every output stream, and records live server and GPU-memory logs:

RUN_ROOT=/path/to/run-root \
MODEL_ROOT=/path/to/MiniMax-H3 \
GPU_IDS=0,1 \
PROFILE=rtx5090 \
bash recipes/MiniMaxAI/scripts/run_h3_2gpu_all_tasks.sh

Use PROFILE=rtx4090 for the 24 GB defaults. The script fails before inference when a checkpoint partition, idle GPU, host-RAM budget, or shared FFmpeg/TorchCodec dependency is missing.

Launch — four GPUs (validated best practice)

Measured on 4× NVIDIA B300: no CPU or layerwise offload, Ulysses SP degree 4, native tiled VAE patch parallelism degree 4, regional torch.compile over the repeated DiT blocks, FlashAttention, Ring and TP left at 1.

CUDA_VISIBLE_DEVICES=0,1,2,3 \
FLASHINFER_DISABLE_VERSION_CHECK=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
vllm serve /path/to/MiniMax-H3/FL2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --num-gpus 4 \
  --usp 4 \
  --ring 1 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
  --vae-use-tiling \
  --diffusion-attention-backend FLASH_ATTN

Swap the served path to /path/to/MiniMax-H3/Ref2VA and restart to handle Ref2VA requests instead.

RTX PRO 5000 Blackwell — four PCIe GPUs

Select the RTX PRO 5000 Blackwell 4x hardware profile for the validated BF16, no-offload topology: TP2 × Ulysses2, Ring1, text-encoder TP4, VAE patch parallel 4 in tile mode, and CUDNN_ATTN. On the measured dual-socket host, CUDA_VISIBLE_DEVICES=0,2,1,3 kept the TP pairs on local PXB paths; check nvidia-smi topo -m and adapt the order on other systems.

A five-step topology screen after warmup produced decodable H.264 video with stereo AAC:

Workload	Client E2E	Encode	Denoise	VAE decode	Peak/GPU
T2VA, 124 frames, 1344×768	32.993 s	0.034 s	27.915 s	4.307 s	67.6 GiB
FL2VA first frame, same output shape	30.130 s	0.276 s	24.396 s	4.339 s	67.6 GiB

These are topology-screening results, not 50-step throughput claims. Nsight Systems aggregation for the same run was:

Workload	AllGather	Send/Recv	NCCL other	NCCL total	Dense FMHA	max-min/mean
T2VA	0.93%	8.53%	14.63%	24.09%	31.15%	1.11%
FL2VA	0.89%	7.26%	15.76%	23.91%	32.53%	0.13%

NCCL other covers NCCL kernels outside recognized AllGather and point-to-point Send/Recv, including AllReduce, ReduceScatter, or Broadcast when present.

AMD ROCm — four MI300X GPUs

Verified on 4× AMD Instinct MI300X (gfx942) with the ROCm H3 image, AITER packed variable-length attention, Ulysses 4, text-encoder TP4, and native tiled VAE patch parallelism 4:

docker pull vllm/vllm-omni-rocm:minimax-h3

HIP_VISIBLE_DEVICES=0,1,2,3 \
VLLM_ROCM_USE_AITER=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
vllm serve /path/to/MiniMax-H3/FL2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --num-gpus 4 \
  --usp 4 \
  --ring 1 \
  --text-encoder-tp-size 4 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
  --vae-use-tiling \
  --diffusion-attention-backend FLASH_ATTN

On ROCm, FLASH_ATTN resolves to AITER on supported Instinct architectures. Do not install the CUDA-only [fa4] extra or set FLASHINFER_DISABLE_VERSION_CHECK.

The refreshed vllm/vllm-omni-rocm:minimax-h3 tag includes the Ref2VA soundfile fallback, TorchCodec, and FFmpeg, so T2VA, FL2VA, and the complete Ref2VA path use the same official image.

AMD ROCm — four MI355X GPUs

Verified on 4× AMD Instinct MI355X. The official ROCm image includes the MiniMax H3 Ref2VA fixes, TorchCodec, and FFmpeg. We tested both Ulysses 4 with Ring 1 and Ulysses 1 with Ring 4. The command below shows the Ulysses 4 / Ring 1 configuration with tiled VAE patch parallelism degree 4, AITER, and CPU offload:

docker pull vllm/vllm-omni-rocm:minimax-h3

HIP_VISIBLE_DEVICES=0,1,2,3 \
VLLM_ROCM_USE_AITER=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
vllm serve /path/to/MiniMax-H3/Ref2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --num-gpus 4 \
  --usp 4 \
  --ring 1 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
  --vae-use-tiling \
  --enable-cpu-offload \
  --text-encoder-tp-size 4

Replace Ref2VA with FL2VA to serve text-to-video+audio or first-frame-to-video+audio.

Three constraints on this configuration:

• Do not add --enforce-eager. Regional compile is the default (--diffusion-compile-granularity regional) and is worth ~9.5%. The first request includes compilation — warm the server once before measuring steady state.
• --cfg-parallel-size must stay 1. H3 is CFG-distilled and has no negative branch; anything greater is rejected with an explicit error rather than silently duplicating a branch that does not exist.
• The VAE supports its native tile mode only — not spatial_shard_width or spatial_shard_height (those belong to the distributed Wan autoencoder). Patch parallel size must be 1 or the full DiT group size.

Launch — single GPU (accuracy and memory first)

Model-level CPU offload keeps the Qwen3-VL encoder and the DiT from being resident simultaneously. This is the accuracy-qualified reference path; it trades PCIe/NVLink transfer latency for a much smaller GPU footprint, and needs enough system RAM for the offloaded components.

CUDA_VISIBLE_DEVICES=0 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
vllm serve /path/to/MiniMax-H3/FL2VA \
  --omni \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --num-gpus 1 \
  --enable-cpu-offload \
  --diffusion-attention-backend FLASH_ATTN

Text-encoder tensor parallelism

By default the whole Qwen3-VL encoder is resident on the DiT main rank, making it the peak-memory hotspot on multi-GPU no-offload runs. --text-encoder-tp-size N shards it across the first N DiT ranks using vLLM-style tensor-parallel layers on a dedicated encoder process group:

vllm serve /path/to/MiniMax-H3/FL2VA \
  --omni --trust-remote-code \
  --num-gpus 4 --usp 4 --ring 1 \
  --text-encoder-tp-size 4 \
  --vae-patch-parallel-size 4 --vae-parallel-mode tile --vae-use-tiling \
  --diffusion-attention-backend FLASH_ATTN

• N must divide the Qwen3-VL head counts (64 attention heads / 8 KV heads) → 1, 2, 4 on a 4-GPU server; 1, 2, 4, 8 on 8 GPUs.
• On 4 GPUs, N=4 drops the per-rank engine peak from 133 GB to 103 GB (~35.7 GB) at no measurable throughput cost, while each non-main rank gains ~51.5/N GB.
• Numerically: N=1 reproduces the HF reference path bit-exactly (max_abs = 0); N=4 introduces bounded BF16 rounding only (row-parallel all-reduce in FP32), giving 31.11 dB PSNR / 0.9566 SSIM end-to-end against N=1 with identical structure.
• Combining N>1 with regional compile used to produce a ValueError: v must be finite on the second request. That is fixed — the token refiner's attention runs on replicated rows before sp_prepare and was wrongly getting a Ulysses SP all-to-all.

Key request parameters

Parameter	Recommended	Notes
task	t2va, fl2va, ref2va	Passed inside extra_params; must match the served partition
duration	4–15 s	Decimal seconds in extra_params; snapped to H3's legal 17n+5 frame count (15 s → 362 frames / 15.083 s)
fps	24	Output FPS is fixed
num_inference_steps	50	Matches the reference accuracy workloads
flow_shift	12	Video sigma shift
audio_flow_shift	3	Audio sigma shift, in extra_params
width, height	see below	Server-side validation: multiples of 32, aspect ratio 1:4 to 4:1
seed	fixed value	Output is deterministic per request at a fixed seed

On width/height, the serving-side check is broader than the documented product modes — pick a short edge of 768 or 1440 in one of the supported ratios (21:9, 16:9, 4:3, 1:1, 3:4, 9:16) to stay on shapes the model was trained and validated for. FL2VA can omit both and inherit the first frame's aspect ratio at a 768 px short edge. Prompts cap at 7,000 characters.

Use POST /v1/videos/sync when you want the MP4 in the response body; POST /v1/videos is the async job-polling variant.

Validated performance (4×B300)

Workload	Config	Result
FL2VA, 209 frames, 1248×768 (8.7 s)	no offload, U4, VPP4 tile, regional compile	86.96 s client E2E (±0.23 s)
Two-video Ref2VA, 362 frames, 1344×768 (15 s)	same	784.4 s accounted model-stage mean

Stage split for the FL2VA case: text encoder 0.21 s, visual encoder 0.20 s, DiT 79.1 s (88% of the request), video+audio VAE decode 2.40 s. VAE patch parallelism is the cheapest win available — it cuts decode 3.4–3.5× (8.24 s → 2.40 s).

Validated performance (4×MI300X)

Measured with vllm/vllm-omni-rocm:minimax-h3 (sha256:29d1946af9c69e3e0a7128c247bdc8c82437ead43fbc366fed443533cf6ce9e8), BF16, AITER, U4, text-encoder TP4, VPP4 tile, regional compile, one warmup, and the synchronous Video API:

Workload	Model stages	Client E2E
T2VA, 209 frames, 1344×768 (8.7 s)	encode 0.09 s, denoise 244.04 s, decode 4.15 s	267.42 s
FL2VA, 209 frames, 1344×768 (8.7 s)	encode 13.98 s, denoise 257.58 s, decode 4.11 s	287.07 s

Both outputs were validated as 209-frame H.264 at 24 FPS with 32 kHz stereo AAC. A v0.26.0 source build additionally completed image+audio and video-reference Ref2VA, plus a 2048×1088 T2VA run. These measurements describe tested shapes, not a general latency guarantee.

A with_stack profile of the two-video Ref2VA case shows the pipeline is GPU-attention-bound, not host-starved: FlashAttention-4 is ~76% of diffuse device time, Ulysses send/recv is secondary, GPU utilization is 92.8–94.3% and the CPU-idle union inside the transformer forwards is 3.15%. Reducing DiT time further needs a change in attention complexity (reference-latent pooling/pruning, block-sparse attention), which is not lossless.

Against the checkpoint's own reference implementation, the accuracy-qualified path scores SSIM 0.9873–0.9896, PSNR 39–42 dB, pixel cosine 0.9996+, and audio log-mel cosine 0.977–0.996 on T2VA/FL2VA. The two-video Ref2VA composite scores lower on raw pixels (SSIM 0.628) largely because of H.264 bitrate differences, while CLIP cosine is 0.9816 and audio log-mel cosine 0.9869 — content and motion stay aligned.

Cache-DiT

Opt-in only. A 50-step T2VA probe cut denoise time 30.2% (121.01 s → 84.47 s) but dropped SSIM to 0.831 against the uncached output. H3's video trajectory is sensitive to cached denoise steps — choose the cache budget against your own quality target.

Offload and accuracy

On a single GPU, model-level CPU offload is the accuracy-qualified path. On four GPUs, no-offload is both faster and accuracy-clean once the encoder stays resident: an earlier no-offload build performed a ~63 GiB DiT GPU→CPU→GPU round trip inside encode_prompt, which was pure overhead (encoder 30.38 s → 0.207 s once removed, ~146×).

Known limitations

• Each server process loads exactly one checkpoint partition.
• H3 executes one generation request per diffusion batch today.
• The first request is a compile warmup only on the resident regional-compile profiles. The two-GPU DLO recipes use --enforce-eager and do not compile the DiT.
• The serving path accepts fewer references than the model supports. H3 documents up to 9 images, 3 video clips, and 3 audio clips (12 files) per Omni Reference request; the current vLLM-Omni path takes exactly one image plus one audio reference, or one or more videos with no separate audio_reference (it uses the source soundtracks).
• The 768 px short-edge mode is available for T2VA and FL2VA; 1344×768 is the documented 16:9 request shape. The 24 GB DLO recipe uses 1024×576 as its lower-memory starting point.
• --cfg-parallel-size > 1 is rejected by design (CFG-distilled, no negative branch).
• The VAE supports the native tile parallel mode only.
• A U2 × Ring2 hybrid currently fails with an attention-mask length mismatch; use pure Ulysses.
• FP8 quantization is not supported yet — a transformer-side blocker is known and deferred past the day-0 release.
• Pure Ulysses still replicates the 66.3 GB DiT on every rank, so 64 GB GPUs cannot use --usp N --tp 1 as a resident capacity path. Use DiT TP plus DLO (the two-GPU recipes above), or model-level CPU offload; text-encoder TP alone is not sufficient.
• DLO is a capacity path, not a free speedup: it needs substantial host RAM and streams weights over PCIe on every denoise step. Increase --dlo-resident-layers only after checking peak HBM on the exact request shape.

References

• Model card
• vLLM-Omni Video API
• vLLM-Omni diffusion parallelism
• vLLM-Omni diffusion attention backends
• vLLM-Omni supported models
• vLLM-Omni GPU installation
• ROCm H3 tracking issue