Multi-Token Prediction (MTP) for Faster LLM Inference: How Qwen 3.5 Changes Agent Speed

If you’ve been running LLM agents in production, you already know where the time goes: it’s not the prefill, it’s the decode phase. Every token generated sequentially, one at a time, is a fundamental bottleneck that no amount of hardware throwing solves cleanly. Multi-token prediction is the architectural change that directly attacks this problem — and with Qwen’s roadmap and MLX’s upcoming support, it’s about to move from research curiosity to something you can actually deploy.

This article is about what MTP means practically for agent developers: how it works, what the real latency gains look like, how to measure whether it’s actually helping you, and where the current implementations fall short. If you’re building anything where response time matters — tool-calling agents, real-time assistants, high-throughput pipelines — this is worth understanding now before it lands in every framework changelog.

What Multi-Token Prediction Actually Does (and What It Doesn’t)

Standard autoregressive decoding generates one token per forward pass. The model sees the full context, predicts a probability distribution over the vocabulary, samples one token, appends it, and repeats. It’s simple and it works, but it means latency scales linearly with output length. A 200-token response takes 200 sequential forward passes.

Multi-token prediction changes this by training the model to predict multiple future tokens simultaneously from a single forward pass. The original Meta research (MTP in the LLaMA context) added auxiliary prediction heads on top of the transformer — one head predicts token N+1, another predicts N+2, and so on. During inference, you can use these predictions speculatively.

This is related to but distinct from speculative decoding. In speculative decoding, a small draft model generates candidate tokens that a larger verifier model checks in parallel. MTP bakes the multi-step prediction capability directly into a single model, avoiding the need for a separate draft model entirely. The speedup mechanism is similar — you’re doing parallel verification of candidate tokens — but the implementation complexity is lower because you’re shipping one model, not two.

The Speculative Decoding Connection

When a model trained with MTP is used for inference, you run the main model forward pass and simultaneously get predictions for the next N tokens from the auxiliary heads. You then verify those predictions: if they’re accepted (they match what the main model would have generated), you advance multiple positions at once. If one fails, you fall back to the standard token and restart speculation from there.

The acceptance rate is everything here. If the auxiliary heads are predicting accurately, you might accept 3-4 tokens per forward pass instead of 1, giving you a 3-4x throughput improvement. If acceptance rates are poor (which happens in high-entropy generation like code with variable naming), you’re paying extra compute for minimal gain.

Real-world numbers from the DeepSeek-V3 MTP implementation showed roughly 1.8x throughput improvement in benchmarks, with latency reductions of 40-50% on typical output lengths. That’s not free — the auxiliary heads add parameters and training cost — but for inference-heavy workloads, it’s significant.

Where Qwen Fits Into This Picture

Qwen 3 (and the anticipated 3.5 series) is architecturally positioned to take MTP seriously. The Qwen team has been explicit about adopting inference-time efficiency techniques from DeepSeek’s playbook, and the model architecture already supports the kind of weight-sharing between prediction heads that makes MTP practical without catastrophic parameter bloat.

The key design decision in Qwen’s approach is that MTP heads share transformer block weights with the main model rather than adding entirely new parameters. This means the memory overhead is relatively modest — early estimates suggest around 10-15% additional parameter count for 2-4 token prediction depth, compared to the full model size. For a 7B model running on 24GB VRAM, that’s still within range for single-GPU deployment.

MLX Support and What It Means for Local Deployment

Apple’s MLX framework is adding MTP support, which matters more than it might seem. MLX is the primary way developers run serious local inference on Apple Silicon, and the M-series chips’ unified memory architecture actually makes speculative decoding more attractive than on discrete GPUs — memory bandwidth is less of a bottleneck, so the verification step is cheaper relative to the generation step.

When MLX ships MTP support (currently in active development in the mlx-lm repository), running a Qwen 3.5 variant locally with MTP enabled could genuinely change the calculus for edge deployments. Rough projections based on current MLX speculative decoding benchmarks suggest you might see 60-80 tokens/second on M2 Pro with a 7B MTP model, compared to 35-45 tokens/second with standard decoding. That’s the difference between an agent response feeling snappy and feeling slow.

How This Changes Agent Architectures

The latency improvement isn’t uniform across use cases, which matters a lot for how you should think about your architecture.

Where MTP Wins Hardest

• Short to medium outputs with predictable structure: JSON responses, function call arguments, structured data extraction. These have high acceptance rates because the token distribution is narrow and predictable. You’ll see the best theoretical speedups here.
• Multi-step agent loops: If your agent does 10 tool calls per task, each with a reasoning step plus a structured output, MTP compounds across each step. A 40% latency reduction per step becomes meaningful across a full workflow.
• Streaming interfaces where time-to-first-token matters: MTP doesn’t directly help TTFT, but because subsequent tokens arrive faster, the perceived responsiveness improves.

Where MTP Helps Less

• Long-form creative or reasoning output: High-entropy generation means lower acceptance rates. Writing a detailed analysis or generating novel code paths will see smaller gains than filling in structured templates.
• Very short outputs: If you’re generating 10-20 token responses, the overhead of the speculation mechanism eats into the gains.
• Batch inference on GPUs: The arithmetic changes when you’re running hundreds of parallel requests on A100s. MTP is primarily a latency win; throughput at scale is a different optimization problem better addressed by continuous batching and quantization.

Measuring Whether MTP Is Actually Helping You

Don’t trust benchmark numbers blindly. Here’s how to measure what matters for your specific workload:

import time
import statistics

def benchmark_inference(model_fn, prompts: list[str], runs: int = 3) -> dict:
    """
    Measure real latency for your specific prompt distribution.
    model_fn should return (output_text, token_count) tuple.
    """
    results = []
    
    for prompt in prompts:
        times = []
        token_counts = []
        
        for _ in range(runs):
            start = time.perf_counter()
            output, n_tokens = model_fn(prompt)
            elapsed = time.perf_counter() - start
            
            times.append(elapsed)
            token_counts.append(n_tokens)
        
        avg_time = statistics.mean(times)
        avg_tokens = statistics.mean(token_counts)
        
        results.append({
            "prompt_preview": prompt[:50],
            "avg_latency_s": round(avg_time, 3),
            "avg_tokens": avg_tokens,
            # This is what actually matters for UX
            "tokens_per_second": round(avg_tokens / avg_time, 1),
            # Useful for agent loop cost estimation
            "latency_per_token_ms": round((avg_time / avg_tokens) * 1000, 2),
        })
    
    return {
        "individual": results,
        "mean_tps": statistics.mean(r["tokens_per_second"] for r in results),
        "p95_latency": statistics.quantiles(
            [r["avg_latency_s"] for r in results], n=20
        )[-1],  # 95th percentile
    }

# Use prompts representative of your actual agent workload
# Don't benchmark on "write me a poem" if you're building a data extraction agent
agent_prompts = [
    "Extract the company name, date, and total amount from this invoice: ...",
    "Given these search results, which one answers the user's question about...",
    "Call the appropriate function to handle this user request: ...",
]

The key metric is tokens per second on your actual prompt distribution, not on synthetic benchmarks. A JSON extraction agent and a reasoning chain agent will see completely different MTP acceptance rates.

Setting Up a Baseline Before MTP Lands

Run this benchmark now against your current setup. When MLX ships MTP support or when you switch to a Qwen 3.5 MTP variant, rerun it with identical prompts. That’s your actual speedup number, not the paper’s number.

# Quick integration test for mlx-lm (standard, pre-MTP)
# When MTP support ships, the interface should be identical
# — only the underlying inference changes

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen2.5-7B-Instruct-4bit")

def qwen_inference(prompt: str) -> tuple[str, int]:
    messages = [{"role": "user", "content": prompt}]
    formatted = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )
    
    response = generate(
        model, 
        tokenizer, 
        prompt=formatted,
        max_tokens=256,
        verbose=False  # Set True to see token-by-token for debugging
    )
    
    # mlx-lm returns the full string; approximate token count
    token_count = len(tokenizer.encode(response))
    return response, token_count

Cost Implications at Scale

If you’re running agents against API endpoints rather than local models, MTP matters differently. You pay per token generated, so throughput improvements don’t reduce your bill — but latency reduction does reduce your wall-clock time per task, which affects how many parallel workers you need to hit a given throughput target.

Rough math: if you’re running an agent pipeline that currently takes 8 seconds per task and you need to process 1,000 tasks per hour, you need at least 3 concurrent workers. With a 40% latency reduction from MTP (down to ~5 seconds), you can hit the same throughput with 2 workers. If each worker costs $0.05/hour in infrastructure, that’s marginal — but at 10,000 tasks/hour the compute savings compound.

For local inference, the math is more direct. Running a 7B model on an M2 Pro for agent workloads currently costs roughly the machine’s depreciation and power. Doubling throughput effectively halves your compute cost per task. At 100,000 agent tasks per month, the difference between 40 t/s and 70 t/s is meaningful.

What’s Not Ready Yet and What Will Break

Be honest with yourself about the current state. As of mid-2025:

• MLX MTP support isn’t in the stable release yet. It’s in active development. Don’t build a production dependency on it today; watch the mlx-lm GitHub releases.
• Qwen 3.5 with MTP training isn’t publicly released. The architecture supports it; the trained weights with MTP heads are still forthcoming.
• Acceptance rate logging is immature. Most inference frameworks don’t expose per-request acceptance rates yet, which makes it hard to debug why MTP isn’t helping as much as expected for a specific use case.
• Quantization interacts badly with MTP at aggressive levels. 4-bit quantization of auxiliary prediction heads degrades acceptance rates significantly. 8-bit is probably the floor for getting the advertised speedups.

Who Should Prioritize This Right Now

Solo founders building local-first agents on Apple Silicon: Watch the mlx-lm releases actively. When Qwen 3.5 MTP weights land, test them immediately on your actual workload. This is the single highest-leverage inference improvement coming to your hardware in the near term.

Teams running self-hosted inference on GPU: Start evaluating DeepSeek-V3’s MTP implementation now since it’s already shipped. The patterns transfer directly to Qwen when it arrives. vLLM has speculative decoding support that’s compatible with MTP-trained models.

API-only builders: This is less urgent for you. Your costs are token-based, not compute-based. Focus on output token reduction through better prompting before chasing infrastructure speedups. That said, if you’re latency-sensitive and considering switching to a provider that runs Qwen 3.5 MTP, the response time improvement is real and worth factoring in.

Automation builders on n8n/Make: Multi-token prediction will show up transparently through API latency improvements as providers upgrade their infrastructure. You don’t need to do anything, but you will notice faster LLM nodes in your workflows — especially on structured output steps.

The bottom line: multi-token prediction is one of the few inference improvements that doesn’t require you to sacrifice output quality to get speed. It’s not magic, acceptance rates vary, and the current tooling is still maturing — but the direction is clear. The models and frameworks that ship MTP properly will have a measurable, real-world advantage for agent workloads. Set up your benchmarks now so you can verify the claims when the weights land.