vLLM/Recipes
Thinking Machines Lab

thinkingmachines/Inkling

Natively multimodal 1T-parameter MoE from Thinking Machines Lab — text/image/audio in, text out, up to 1M context — with relative attention, short convolution, and shared expert sinks.

380 tok/s/user with MTP8 (mean acceptance length 4.5) and 140 tok/s/user without MTP on 4× GB200 GPUs

View on HuggingFace
moe975B / 41B1,048,576 ctxvLLM 0.26.0+multimodal
Guide

Overview

TML Inkling is a natively multimodal 1T-parameter Mixture-of-Experts model from Thinking Machines Lab. It accepts text, image, and audio inputs and generates text, with a context length of up to 1M tokens. vLLM supports Inkling on Day 0 with full feature parity — LoRA, speculative decoding (MTP), TP/DP/EP/PP parallelism, prefix caching, and disaggregated serving.

Architectural highlights:

  • Attention. 66-layer decoder backbone — 11 full-attention layers and 55 sliding-window layers — all grouped-query attention with head size 128. Inkling replaces RoPE with relative attention: a learned relative-position term added to the pre-softmax logits.
  • Short convolution (sconv). Each layer applies four window-4 sconv modules (on attention keys, values, output, and MoE output). vLLM manages the sconv cache as the KV cache of a virtual sliding-window layer, so prefix caching works seamlessly, and shards sconv across the channel dimension (reduce-scatter / all-gather) to avoid replicating cache and compute across TP ranks.
  • MoE with expert sinks. 256 routed experts (top-6) plus 2 shared experts — 8 experts per token. The 2 shared experts act as expert sinks: they participate in routing-score computation but are excluded from the top-6 candidate pool.
  • MTP. 8 chained MTP heads (single-layer full-attention transformers with a dense MLP, all BF16) enable up to 9 tokens per forward step for speculative decoding.

Variants

  • NVFP4 (default): Only the routed experts are quantized to NVFP4; shared experts and the qkvr linears remain BF16. This is a mixed-precision checkpoint. Blackwell uses its native FP4 tensor cores; on Hopper, the MoE kernel dequantizes the NVFP4 expert weights to BF16 on the fly.
  • BF16: Full BF16 weights (thinkingmachines/Inkling). Runs on both Hopper (H200&H20) and Blackwell, at the cost of substantially more VRAM.
  • MXFP4 (AMD): lightseekorg/Inkling-MXFP4 keeps the full 1M-token context at TP4 on MI355X and MI300X. MI355X runs native AITER MXFP4 MoE; MI300X uses Triton on-the-fly MXFP4 emulation.

Prerequisites

  • Hardware: NVIDIA Hopper (H200/H20) or Blackwell for NVFP4 and BF16; AMD MI300X, MI325X, or MI355X with TP4 for MXFP4.
  • vLLM: Nightly wheels (Day-0 support just landed).
  • Parallelism: 1T parameters requires multi-GPU. The featured configuration is 4x GB200 (--tensor-parallel-size 4) for the NVFP4 variant. On Hopper, NVFP4 uses TP8 on H200; the BF16 variant needs multi-node.

Client Usage

Launch the server (NVFP4 on 4x GB200):

export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 4 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code

For a multi-node BF16 deployment on Hopper (H200/H20), select the DEP strategy with Nodes = 2 in the command builder above — it renders the per-node vllm serve commands (TP within each node, DP across nodes).

On AMD, select the MXFP4 variant. TP4 is validated with the native 1M context on both GPU families.

Equivalent MI355X launch:

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export INKLING_GFX950_GLUON=0

vllm serve lightseekorg/Inkling-MXFP4 \
  --tensor-parallel-size 4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --moe-backend aiter \
  --block-size 128 \
  --trust-remote-code

Query the server with the OpenAI SDK, including an image input:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)

Performance

vLLM implements Inkling with a series of optimizations: sconv-aware TP sharding (each rank stores only its channel shard of the sconv cache), low-latency fused reduce-scatter / all-gather collectives built on FlashInfer's Lamport-protocol all-reduce, the FA4 sheared-bias attention kernel for relative attention, plus kernel fusion, PDL, and multi-streaming. Enabling MTP speculative decoding adds further per-user throughput.

References